Use an unbounded thread pool for StreamExecutor PjRt clients #36030
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
copybara-service
bot
force-pushed
the
test_853019406
branch
2 times, most recently
from
74513a4 to
7a28b74
Compare
If `tsl::thread::ThreadPool` runs out of threads, it runs the callback inline on the thread that called `Schedule()`. This pattern is known to be prone to deadlocks (e.g., if `Schedule` is called while holding a mutex that the callable needs to acquire, it will deadlock) and may harm the asynchrony in the tail case, so switching to an unbounded thread pool based on `tsl::UnboundedWorkQueue`. Since `tsl::thread::ThreadPool` is not extensible, this requires replacing the executor type from `tsl::thread::ThreadPool` to `AsyncWorkRunner`. The existing thread pool is still kept because `ExecutableBuildOptions::set_compile_thread_pool()` requires `tsl::thread::ThreadPool`. PiperOrigin-RevId: 853352782
copybara-service
bot
force-pushed
the
test_853019406
branch
from
7a28b74 to
f4bab1d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Use an unbounded thread pool for StreamExecutor PjRt clients
If
tsl::thread::ThreadPoolruns out of threads, it runs the callback inline on the thread that calledSchedule(). This pattern is known to be prone to deadlocks (e.g., ifScheduleis called while holding a mutex that the callable needs to acquire, it will deadlock) and may harm the asynchrony in the tail case, so switching to an unbounded thread pool based ontsl::UnboundedWorkQueue.Since
tsl::thread::ThreadPoolis not extensible, this requires replacing the executor type fromtsl::thread::ThreadPooltoAsyncWorkRunner. The existing thread pool is still kept becauseExecutableBuildOptions::set_compile_thread_pool()requirestsl::thread::ThreadPool.Reverts 485e912