Skip to content

Enhancement: Improve test suite performance #123

@dougborg

Description

@dougborg

Summary

The test suite currently takes ~13 minutes to run locally (803 seconds for 250 tests), which impacts developer productivity and CI feedback loops.

Current Performance

  • Local execution: 13 minutes 23 seconds (803.49s)
  • 250 tests (11 xfailed)
  • CI execution: ~2-3 minutes per Python version × 5 versions = 10-15 minutes total

Potential Optimizations to Investigate

1. Parallel Test Execution

  • Use to run tests in parallel
  • Could reduce runtime by 2-4x depending on CPU cores
  • Command: pytest -n auto

2. Test Fixtures Optimization

  • Review fixture scope (function vs module vs session)
  • Cache expensive setup operations (file generation, parsing)
  • Lazy load test data

3. Selective Test Running

  • Implement test markers for fast/slow tests
  • Run quick smoke tests first in CI
  • Full suite only on main branch or nightly

4. Code Generation Caching

  • Cache generated code for identical OpenAPI specs
  • Reduce redundant generation in similar test cases

5. CI-Specific Optimizations

  • Matrix strategy to fail-fast on first failure
  • Split test suite across multiple jobs
  • Use GitHub Actions cache for dependencies

Success Criteria

  • Reduce local test runtime to < 5 minutes
  • Reduce CI feedback time to < 2 minutes per Python version
  • Maintain 90%+ code coverage
  • No flaky tests introduced

Non-Goals

  • Don't sacrifice test coverage for speed
  • Don't introduce test interdependencies
  • Don't skip important integration tests

Priority

Medium - Developer quality of life improvement, not blocking


Discovered during #98 refactor work where full test suite verification took significant time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions