[Feature]: Baseline Scoring for detection regressions #157
Description
Summary
Baseline Scoring for detection regressions in fitness score and health checks
Description
The feature would enable end-users to differentiate between cluster performance when running an actual krkn-ai test vs when no tests.
Background and Context
No response
Current Code Analysis
No response
Issues Identified
No response
Problem
Already covered in the description.
Proposed Solution
Changes in Krkn-AI main loop
This would be the new flow of execution:
- User trigger Krkn-AI
- Run baseline:
- For certain duration run the dummy scenario.
- Capture health checks and fitness score for this dummy scenario (call it baseline)
- Start Generation 1 until end to discover faults with actual krkn-ai tests
Regression Detection
We can use the baseline score and setup some threshold to detect regression with different scenarios. This calculation can take place after running each scenario.
Configurable
- Duration for running dummy baseline scenario
Implementation Steps
No response
Acceptance Criteria
No response
Testing Checklist
No response
Benefits
No response
Alternatives Considered
No response
Related Documentation
No response
Additional Context
No response