-
Notifications
You must be signed in to change notification settings - Fork 42
Description
Summary
The JSONSummaryReporter class (krkn_ai/reporter/json_summary_reporter.py) currently has zero unit test coverage. Both existing reporters GenerationsReporter and HealthCheckReporter have dedicated test files in tests/unit/reporter/, but test_json_summary_reporter.py does not exist.
Description
The JSONSummaryReporter was recently added to generate a unified results.json summary of Krkn-AI experiment runs. It consolidates run metadata, config summaries, fitness statistics, top-10 best scenarios, and per-generation fitness progression into a single file.
Despite being a critical component of the reporting pipeline (called from GeneticAlgorithm.save() at the end of every run), this reporter has no test coverage. Adding comprehensive unit tests will ensure correctness, prevent regressions, and maintain parity with the test coverage of the other two reporters.
Background and Context
The existing reporter test suite follows a consistent pattern:
tests/unit/reporter/test_generations_reporter.py— 4 tests covering YAML/JSON output and graph generationtests/unit/reporter/test_health_check_reporter.py— 7 tests covering CSV reports, graph plotting, and fitness CSV writing
Both use shared fixtures from tests/conftest.py (temp_output_dir, mock_cluster_components) and construct test data using DummyScenario and CommandRunResult models. The new test file should follow the same conventions.
Current Code Analysis
JSONSummaryReporter (in krkn_ai/reporter/json_summary_reporter.py) has 3 public/private methods that need testing:
-
generate_summary()— Builds the full results dict with: run_id, seed, timestamps, duration, config summary, statistics (total/unique scenarios, best/avg fitness), best_scenarios list, and fitness_progression list. -
_build_fitness_progression()— Iteratesbest_of_generationto create per-generation best/average fitness data. -
_build_best_scenarios()— Sortsseen_populationby fitness score and returns the top 10 with rank, scenario_id, generation, fitness_score, scenario_type, and parameters. -
save()— Writesgenerate_summary()output toresults.json.
Currently no file tests/unit/reporter/test_json_summary_reporter.py exists.
Issues Identified
- Zero test coverage for JSONSummaryReporter — any regression will go undetected
- No validation that
generate_summary()produces the expected JSON structure - No edge-case testing (empty populations, single generation, zero fitness scores)
- No verification that
save()writes valid, parseable JSON to the correct path - Test coverage gap compared to the other two reporters which are well-tested
Problem
Without tests, changes to data models (CommandRunResult, FitnessResult, BaseScenario) or the reporter itself could silently break results.json generation. Since results.json is intended to be the primary machine-readable summary for downstream tools (including the planned interactive visualization from Issue #74), its correctness is critical.
Proposed Solution
Create a new test file tests/unit/reporter/test_json_summary_reporter.py following the same patterns as the existing reporter tests:
- Use
temp_output_dirandDummyScenariofixtures fromconftest.py - Construct
CommandRunResultobjects with known fitness scores - Build
seen_populationdicts andbest_of_generationlists as test inputs - Assert on the structure and values of the generated summary dict
- Verify file output from
save()
Implementation Steps
No response
Acceptance Criteria
No response
Testing Checklist
No response
Benefits
No response
Alternatives Considered
No response
Related Documentation
No response
Additional Context
No response