Skip to content

fix(base): pad predict_proba to the schema's class count - #399

Open
tachyonicClock wants to merge 1 commit into
adaptive-machine-learning:mainfrom
tachyonicClock:fix/predict-proba-pad-to-schema
Open

fix(base): pad predict_proba to the schema's class count#399
tachyonicClock wants to merge 1 commit into
adaptive-machine-learning:mainfrom
tachyonicClock:fix/predict-proba-pad-to-schema

Conversation

@tachyonicClock

Copy link
Copy Markdown
Collaborator

Problem

MOAClassifier.predict_proba can return an array shorter than schema.get_num_classes().

stream = CovtypeTiny()                      # 7 classes
learner = HoeffdingTree(stream.schema, grace_period=1)
# ...train on 3 instances (labels 4, 4, 1)...
learner.predict_proba(stream.next_instance())
# array([0., 0.33333333, 0., 0., 0.66666667])   <- length 5, want 7

MOA's getVotesForInstance sizes its return array by the classes seen so far, not by the
schema. Votes are indexed by class index, so the missing entries are always trailing
zeros — this is truncation, not reordering.

Verified against the current moa.jar: 19 of 25 classifiers in capymoa.classifier
returned a short array after 20 training instances on CovtypeTiny. The 6 that looked
correct were the scikit-learn and torch backed learners, which don't go through
MOAClassifier. So this is one bug in the shared wrapper, not a per-learner bug.

predict is unaffected (argmax over a truncated array still gives the right index), and
the evaluators build their own vote template. Only predict_proba callers see it.

Refs adaptive-machine-learning/backlog#96

Changes

  1. Pad in MOAClassifier.predict_proba — zero-pad votes to
    schema.get_num_classes() before the total-is-usable check and the normalisation.
    Pads only when short; a longer array is left alone. The existing reject rules (empty,
    non-finite total, total <= 0) are unchanged — padding an empty array still yields all
    zeros, so the "no prediction" case still returns None.

  2. Delete the duplicate override in DynamicEnsembleMemberSelection — it hand-rolled
    the same padding, which is why it was one of the 6 that looked correct. It also carried
    the old total <= 1e-2 reject rule that was deliberately removed from the base class
    (see test_optimise_flag_does_not_change_results). Now dead weight.

  3. Fix the existing unit test's fake classifiertest_predict_proba_only_rejects_absent_predictions
    builds via MOAClassifier.__new__ and set only moa_learner, so reading self.schema
    would raise. It now gets a fake schema reporting 2 classes, matching its 2-element vote
    cases. No getattr guard was added to the implementation to accommodate the test.

  4. Regression test across all classifiers — the existing shape assertion in
    test_classifiers only passed because it uses ElectricityTiny (2 classes) after a
    full prequential run, by which point both classes have been seen. Added a subtest using
    a fresh learner on RandomTreeGenerator(num_classes=7) (no dataset download) that
    asserts shape (7,) and that values sum to 1, skipping when predict_proba returns
    None since abstaining is allowed. Reuses test_case.learner_constructor, so every
    registered classifier is covered.

Upstream

The underlying bug is in MOA itself: HoeffdingTree.getVotesForInstance delegates to the
leaf's getClassVotes, which returns observedClassDistribution.getArrayCopy() — a
DoubleVector trimmed to the highest class index observed. Notably the treeRoot == null
branch a few lines below does size correctly with inst.dataset().numClasses().

moa.jar is a prebuilt binary pulled from the URL in invoke.yml, so the Java fix can't
land here. A bug report for Waikato/moa has been drafted separately. The CapyMOA padding
is worth keeping either way, since it covers every MOA learner rather than one class.

Verification

  • The issue's snippet now prints a length-7 array summing to 1.
  • Swept every classifier in capymoa.classifier on CovtypeTiny after 20 instances: all
    25 return length 7. Before the fix, 19 returned 5.
  • pytest tests/test_classifiers.py tests/test_evaluation.py tests/test_ssl_classifiers.py
    79 passed, 1 skipped, 52 subtests passed.
  • pytest tests — 290 passed, 25 skipped.
  • pytest --doctest-modules src — 129 passed.
  • invoke fmt — no changes.

🤖 Generated with Claude Code

MOA's getVotesForInstance sizes its return array by the classes seen so
far during training, not by the schema, so MOAClassifier.predict_proba
could silently return an array shorter than schema.get_num_classes().
Votes are indexed by class index, so the fix is to zero-pad trailing
entries rather than reorder anything.

Also removes DynamicEnsembleMemberSelection's duplicate predict_proba
override, which hand-rolled the same padding but kept a stale
total <= 1e-2 reject rule that was deliberately dropped from the base
class.

Fixes adaptive-machine-learning/backlog#96

Assisted-by: claude-code:claude-sonnet-5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant