Skip to content

Design protocol classes and update architecture for model comparison analysis#110

Draft
Copilot wants to merge 5 commits intomainfrom
copilot/fix-109
Draft

Design protocol classes and update architecture for model comparison analysis#110
Copilot wants to merge 5 commits intomainfrom
copilot/fix-109

Conversation

Copy link
Contributor

Copilot AI commented Jun 9, 2025

This PR introduces model comparison analysis capabilities to tab-right by implementing two new protocol classes that follow the existing architectural patterns.

Changes Made

New Protocol Classes

1. Prediction Calculation Protocol (model_comparison_protocols.py)

  • PredictionCalculationP: Protocol for calculating pointwise errors between multiple prediction datasets and true labels
  • Initialized with DataFrame and label column name
  • __call__ method accepts list of prediction Series and optional error function
  • Calculates pointwise error for each prediction against the label using specified or default error function

2. Plotting Protocol (model_comparison_plot_protocols.py)

  • ModelComparisonPlotP: Protocol for visualizing comparison results between multiple predictions and labels
  • Provides interfaces for error distribution plots, pairwise comparisons, and performance summaries
  • Follows existing plotting protocol patterns with Figure type unions (matplotlib/plotly)

Architecture Integration

  • Protocols follow the established pattern used by segmentation and drift analysis modules
  • Uses @runtime_checkable and @dataclass decorators for consistency
  • Maintains the same documentation and type hinting standards
  • Added to API documentation for discoverability

Testing

  • Created comprehensive test suite with 9 new tests following existing patterns
  • Added protocol compliance tests with dummy implementations
  • All 145 tests pass, ensuring no regression in existing functionality
  • Tests verify protocol interfaces, attributes, and method signatures

Example Usage

from tab_right.base_architecture.model_comparison_protocols import PredictionCalculationP
from tab_right.base_architecture.model_comparison_plot_protocols import ModelComparisonPlotP

# Protocol defines interface for comparing multiple model predictions
calc: PredictionCalculationP = MyModelComparison(df, "target_column")
results = calc([pred_series_1, pred_series_2], error_func=custom_metric)

# Protocol defines interface for visualizing comparisons  
plotter: ModelComparisonPlotP = MyComparisonPlotter(calc)
fig = plotter.plot_error_distribution([pred_series_1, pred_series_2])

The architecture now supports extensible model comparison analysis alongside existing segmentation and drift detection capabilities, providing a foundation for future analysis modules.

Fixes #109.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: eh-main-bot <171766998+eh-main-bot@users.noreply.github.com>
Copilot AI changed the title [WIP] Design protocol classes and update architecture for model comparison analysis Design protocol classes and update architecture for model comparison analysis Jun 9, 2025
Copilot AI requested a review from EH-MLS June 9, 2025 15:02
Copilot finished work on behalf of EH-MLS June 9, 2025 15:02
@codecov
Copy link

codecov bot commented Jun 9, 2025

Codecov Report

Attention: Patch coverage is 0% with 23 lines in your changes missing coverage. Please review.

Project coverage is 92.88%. Comparing base (4c432a1) to head (09cd69c).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...se_architecture/model_comparison_plot_protocols.py 0.00% 14 Missing ⚠️
...ht/base_architecture/model_comparison_protocols.py 0.00% 9 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #110      +/-   ##
==========================================
- Coverage   95.78%   92.88%   -2.91%     
==========================================
  Files          19       21       +2     
  Lines         736      759      +23     
==========================================
  Hits          705      705              
- Misses         31       54      +23     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@EH-MLS EH-MLS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A figure object containing the error distribution comparison.

"""
...
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
...

A figure object containing the pairwise comparison visualization.

"""
...
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
...

A figure object containing the performance summary visualization.

"""
...
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this file for now

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed test file as requested in 3b8c1a4

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this file for now

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed test file as requested in 3b8c1a4

Co-authored-by: eh-main-bot <171766998+eh-main-bot@users.noreply.github.com>
Copilot finished work on behalf of EH-MLS June 9, 2025 15:35
@EH-MLS
Copy link
Collaborator

EH-MLS commented Jun 9, 2025

@copilot
ci faling

Copy link
Contributor Author

Copilot AI commented Jun 9, 2025

@copilot
ci faling

Fixed CI failures by resolving linting and type checking issues. All pre-commit hooks, tests (136/136), MyPy type checking, and documentation builds now pass. Commit: 09cd69c

Copilot finished work on behalf of EH-MLS June 9, 2025 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Design protocol classes and update architecture for model comparison analysis

2 participants