Skip to content

Conversation

@mattgodbolt
Copy link
Member

Summary

This PR adds parallelization to the library builder's discovery phase, which is currently the slowest part when checking thousands of library configurations.

Changes

  • Split discovery phase from building phase to enable parallelization
  • Add ThreadPoolExecutor with configurable worker count
  • Use thread-local storage for HTTP sessions (one per worker)
  • Add --parallel-discovery CLI flag (default: 4 workers)
  • Apply same pattern to Fortran and Rust builders
  • Proper constructor parameters instead of attribute injection

Performance Impact

  • Discovery phase should see ~3-6x speedup with 4-8 workers
  • Overall runtime improvement of ~2-3x for library builds
  • Better CPU utilization on expensive GitHub Actions runners

Testing

To test with different worker counts:

# Sequential (old behavior)
ce build --parallel-discovery 1 libraries/c++/fmt/trunk

# Default parallel (4 workers)
ce build libraries/c++/fmt/trunk

# More parallelism
ce build --parallel-discovery 8 libraries/c++/fmt/trunk

Notes

  • Conservative default of 4 workers to avoid overwhelming the conan server
  • Each worker maintains its own HTTP session for connection pooling
  • Thread-safe caching with locks for shared dictionaries
  • No changes to the actual build execution (remains sequential for safety)

Fixes the issue where library discovery takes extremely long on powerful machines that aren't utilizing their CPUs effectively.

- Split discovery phase from building phase to enable parallelization
- Add ThreadPoolExecutor with configurable worker count (default: 4)
- Use thread-local storage for HTTP sessions to avoid connection overhead
- Add --parallel-discovery CLI flag to control worker count
- Apply same pattern to Fortran and Rust builders

This should significantly speed up the 'finding work to do' phase which
currently processes thousands of configurations sequentially.
- Replace instance-level HTTP session with thread-local property
- Each thread now gets its own requests.Session() to avoid race conditions
- Add proper locking for needs_uploading counter modifications
- Optimize lock usage to avoid holding during expensive operations
- Apply same pattern to Fortran and Rust builders for consistency

The previous implementation mixed thread-local and instance sessions,
which could cause crashes or incorrect responses under parallel execution.
@mattgodbolt mattgodbolt marked this pull request as draft September 16, 2025 15:32
@mattgodbolt
Copy link
Member Author

drafting til I land a common code extraction

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants