Performance Regression from 1.45.1 to 1.46 #7692
Description
When initially updating Tokio from 1.45 to 1.47 (later problem bisected to PR: #7342), I experienced a very large performance regression in a custom thread pool designed to process CPU heavy tasks.
The tread pool's code is fully available here.
A high level overview:
- The thread pool manages N threads (
std::thread::) - Tasks between threads are shared with a
flumempmc channel, which is bounded toN*2. - Each thread invokes Tokio's
Runtime::block_on - Each thread runs an async task which:
- Polls a list of tasks stored in an
FuturesUnordereduntil theFuturesUnorderedreturnsPending - Polls a task from the flume channel
- Inserts the task into the
FuturesUnroderedand starts again
- Polls a list of tasks stored in an
Profiles taken (with perf):
- Good (1.45): https://share.firefox.dev/3INrj08
- Bad (1.47): https://share.firefox.dev/3WK0MUw
Unfortunately I did not get a small reproduction done, I bisected tokio commits and load testing the full binary.
A random guess, there are now spurious (or constant) wake ups. As it looks like the pool still gets all of its work done, but when it should be idle it is burning through CPU.
Real Data
Bisected Tokio commits (in the commit message) and the commit hash of the associated builds:
Transition from one version to the next:
CPU Usage:
Internal Service and Thread Pool Utilization (measured on Time spent polling tasks), the top two lines are affected thread pools:
Only Thread pool utilization (again measured on each thread time spent in poll over time), two thread pools:
Average time spent per task in the bigger thread pool does not change (ignore the small section that increases, this is normal). Conclusion from this, time spent is not because the tasks changed:
