Skip to content

Remove redundant learning curve implementation paths #963

Merged
bruAristimunha merged 13 commits intodevelopfrom
unify-split-logic
Feb 4, 2026
Merged

Remove redundant learning curve implementation paths #963
bruAristimunha merged 13 commits intodevelopfrom
unify-split-logic

Conversation

@bruAristimunha
Copy link
Collaborator

@bruAristimunha bruAristimunha commented Feb 3, 2026

Summary

This PR unifies the learning curve implementation by removing redundant code paths and consolidating all learning curve functionality into the LearningCurveSplitter class. This makes learning curves available across all evaluation types (WithinSession, CrossSession, CrossSubject) through a consistent API.

Changes

New Features

  • Add cv_class and cv_kwargs parameters to all evaluation classes for custom cross-validation strategies
  • Implement LearningCurveSplitter as a dedicated sklearn-compatible cross-validator for learning curves
  • Learning curve results now automatically include "data_size" and "permutation" metadata columns

API Changes

  • Breaking: Remove data_size and n_perms parameters from WithinSessionEvaluation
  • Use cv_class=LearningCurveSplitter with cv_kwargs=dict(data_size=..., n_perms=...) instead

Code Health

  • Centralize CV resolution in BaseEvaluation with new _resolve_cv() method
  • Remove redundant methods from WithinSessionEvaluation:
    • get_data_size_subsets()
    • score_explicit()
    • _evaluate_learning_curve()
  • Simplify evaluate() to always use unified _evaluate() path

Other Improvements

  • Add single-class safeguard for LearningCurveSplitter (skip splits where training set collapses to single class)
  • Fix ArrowStringArray shuffle warnings by converting to numpy arrays
  • Update all examples and tests to use new API

Migration Guide

Before:

evaluation = WithinSessionEvaluation(
    paradigm=paradigm,
    datasets=datasets,
    data_size={"policy": "ratio", "value": [0.2, 0.4, 0.6, 0.8, 1.0]},
    n_perms=[5, 5, 5, 5, 1],
)

After:

from moabb.evaluations.splitters import LearningCurveSplitter

evaluation = WithinSessionEvaluation(
    paradigm=paradigm,
    datasets=datasets,
    cv_class=LearningCurveSplitter,
    cv_kwargs=dict(
        data_size={"policy": "ratio", "value": [0.2, 0.4, 0.6, 0.8, 1.0]},
        n_perms=[5, 5, 5, 5, 1],
    ),
)

Test Plan

  • All existing tests pass
  • Learning curve examples work with new API
  • Documentation updated in whats_new.rst

Closes #962

Copilot AI and others added 4 commits February 3, 2026 23:08
* Initial plan

* Remove redundant learning curve implementation

- Remove data_size and n_perms parameters from WithinSessionEvaluation
- Remove get_data_size_subsets(), score_explicit(), and _evaluate_learning_curve() methods
- Simplify evaluate() to always use _evaluate() path
- Update docstring to recommend cv_class=LearningCurveSplitter
- Update all examples to use new LearningCurveSplitter API
- Update tests to use new API

Co-authored-by: bruAristimunha <[email protected]>

* Fix pre-commit linting issues

- Apply black formatting (line length 90)
- Fix import sorting with isort
- Remove unused imports (Optional, StratifiedShuffleSplit)
- Apply ruff fixes

Co-authored-by: bruAristimunha <[email protected]>

* Apply final black formatting fixes

- Remove extra blank lines per black style guide

Co-authored-by: bruAristimunha <[email protected]>

* Fix isort import ordering

- Apply isort to properly order imports in all changed files

Co-authored-by: bruAristimunha <[email protected]>

* Resolve black/isort formatting conflict

- Apply black formatting after isort
- Remove extra blank line after imports

* Add single-class safeguard for LearningCurveSplitter

- Skip splits where training set collapses to single class
- Log warning when splits are skipped due to single-class training sets
- Fix ArrowStringArray shuffle warnings by converting to numpy arrays
- Update tests to call process() since validation happens at evaluation time
- Fix isort import ordering in learning curve examples

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: bruAristimunha <[email protected]>
Co-authored-by: Bru <[email protected]>
Document the learning curve and splitter improvements:
- cv_class and cv_kwargs parameters for all evaluation classes
- LearningCurveSplitter for sklearn-compatible learning curves
- Removal of data_size and n_perms from WithinSessionEvaluation
- Automatic metadata columns for learning curve results
- Centralized CV resolution with _resolve_cv() method
- Removal of redundant learning curve methods
Test that LearningCurveSplitter can be used as cv_class parameter
for all main splitters: WithinSessionSplitter, WithinSubjectSplitter,
CrossSessionSplitter, and CrossSubjectSplitter.
@bruAristimunha bruAristimunha changed the title Remove redundant learning curve implementation paths (#962) Remove redundant learning curve implementation paths Feb 3, 2026
@bruAristimunha bruAristimunha enabled auto-merge (squash) February 4, 2026 16:01
Document the extraction of _load_data() and _get_nchan() helpers into
BaseEvaluation, the move of _pipeline_requires_epochs() to utils.py,
and the WithinSessionSplitter creation hoisted outside the session loop.
@bruAristimunha bruAristimunha merged commit e97f926 into develop Feb 4, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants