Fix CohenKappa ignoring sample weights#1956
Open
chuenchen309 wants to merge 1 commit into
Open
Conversation
CohenKappa.get() divided the observed-agreement and expected-agreement terms by cm.n_samples (the raw observation count) instead of cm.total_weight (the sum of sample weights), so any non-unit sample weight produced a wrong score. Accuracy, which the code comment says it mirrors, already uses total_weight. With all weights equal to 1 the two are identical, which is why this went unnoticed. Matches sklearn.metrics.cohen_kappa_score(..., sample_weight=...). Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
Member
|
Hey there @chuenchen309! May I ask you to open issues instead of immediately proceed to a fix? I appreciate you discovering these bugs, but I would like to dissociate the bug report from the correction. Would that be acceptable for you? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CohenKappa.get()divides the observed agreement (p0) and the two expected-agreement estimators byself.cm.n_samples— the raw, unweighted observation count — while the numerators (total_true_positives,sum_row[c],sum_col[c]) are all weighted sums. When any sample weight differs from 1, the denominator no longer matches the numerator and the returned Kappa is wrong. With unit weightsn_samples == total_weight, which is why the existing tests never caught it.CohenKappainheritsworks_with_weights == True(the base default) and is driven through the documentedupdate(y_true, y_pred, w=...)weight entry point, so it advertises weight support while computing the wrong number. The sibling metrics that genuinely can't handle weights (mutual_info,vbeta,rand,fowlkes_mallows) all overrideworks_with_weights = False; andAccuracy.get()— which the in-line comment says CohenKappa'sp0is "same as" — already divides bytotal_weight.Fix: divide the three agreement terms in
get()byself.cm.total_weightinstead ofself.cm.n_samples.Verification (re-runnable from the diff):
test_cohen_kappa_supports_sample_weightasserts equality withsklearn.metrics.cohen_kappa_score(..., sample_weight=...)and that the weighted result differs from the unweighted one (proving the weight is actually consumed). It is red before the change and green after.river/metrics/suite + doctests: 278 passed.sklearnon the fixed case, the unweighted case, and 500 random weighted fuzz cases (0 mismatches; degenerate single-label windows yieldnanin both and are excluded).This PR was authored by an AI coding agent (Claude Code) running on this account: the AI found the bug, ran the repro, wrote the tests, and wrote this description. The human account holder reviews every change and is accountable for it. The verification above is real and re-runnable from the diff. If this isn't the kind of contribution you want, say so and I'll close it.