-
Notifications
You must be signed in to change notification settings - Fork 36
Open
Description
Summary
The test suite currently takes ~13 minutes to run locally (803 seconds for 250 tests), which impacts developer productivity and CI feedback loops.
Current Performance
- Local execution: 13 minutes 23 seconds (803.49s)
- 250 tests (11 xfailed)
- CI execution: ~2-3 minutes per Python version × 5 versions = 10-15 minutes total
Potential Optimizations to Investigate
1. Parallel Test Execution
- Use to run tests in parallel
- Could reduce runtime by 2-4x depending on CPU cores
- Command:
pytest -n auto
2. Test Fixtures Optimization
- Review fixture scope (function vs module vs session)
- Cache expensive setup operations (file generation, parsing)
- Lazy load test data
3. Selective Test Running
- Implement test markers for fast/slow tests
- Run quick smoke tests first in CI
- Full suite only on main branch or nightly
4. Code Generation Caching
- Cache generated code for identical OpenAPI specs
- Reduce redundant generation in similar test cases
5. CI-Specific Optimizations
- Matrix strategy to fail-fast on first failure
- Split test suite across multiple jobs
- Use GitHub Actions cache for dependencies
Success Criteria
- Reduce local test runtime to < 5 minutes
- Reduce CI feedback time to < 2 minutes per Python version
- Maintain 90%+ code coverage
- No flaky tests introduced
Non-Goals
- Don't sacrifice test coverage for speed
- Don't introduce test interdependencies
- Don't skip important integration tests
Priority
Medium - Developer quality of life improvement, not blocking
Discovered during #98 refactor work where full test suite verification took significant time.
Metadata
Metadata
Assignees
Labels
No labels