Improve cpu utilization with dynamic slice size in doc partitioning #132774
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dnhatn
force-pushed
the
lucene-slice-affinity
branch
2 times, most recently
from
b3f46c6 to
4b52c0e
Compare
dnhatn
changed the title
dnhatn
force-pushed
the
lucene-slice-affinity
branch
3 times, most recently
from
ee117e1 to
2e44fad
Compare
dnhatn
changed the title
dnhatn
changed the title
dnhatn
force-pushed
the
lucene-slice-affinity
branch
2 times, most recently
from
ba09dc3 to
f52e21f
Compare
dnhatn
force-pushed
the
lucene-slice-affinity
branch
from
f52e21f to
1c7de75
Compare
nik9000
reviewed
Aug 15, 2025
Member
nik9000
left a comment
nik9000
left a comment
There was a problem hiding this comment.
Makes a lot of sense to me. So for DOC partitioning this generates more slices by capping the size of slices rather than trying to make as many slices as we have concurrency. And it mitigates the cost of that my having Drivers pluck a slice from the previous segment if possible. Actually - it looks like the way it used to work is that all drivers would try to work on a single segment together and then keep moving. Sort of, concentrating effort. This spreads out which segment is being worked on if possible. I quite like it.
I'm really curious what this does to DOC partitioning performance. The queries could still be a problem, but in the cases where the top level query is empty we kick this in by default.
| * Partitions into dynamic-sized slices to improve CPU utilization while keeping overhead low. | ||
| * This approach is more flexible than {@link #SEGMENT} and works as follows: | ||
| * | ||
| * <p>1. The slice size starts from a desired size based on {@code task_concurrency} but is capped |
| if (currentSlice == null || sliceIndex >= currentSlice.numLeaves()) { | ||
| sliceIndex = 0; | ||
| currentSlice = sliceQueue.nextSlice(); | ||
| currentSlice = sliceQueue.nextSlice(currentSlice); |
| } | ||
| return slices; | ||
| // Cap the desired slice to prevent CPU underutilization when matching documents are concentrated in one segment region. | ||
| int desiredSliceSize = Math.clamp(Math.ceilDiv(totalDocCount, requestedNumSlices), 1, MAX_DOCS_PER_SLICE); |
Member
There was a problem hiding this comment.
I think we shouldn't call this requestedNumSlices any more - it's just taskConcurrency here. At least, we're not respecting the request for the number of slices - we absolutely got above it via MAX_DOCS_PER_SLICE.
Collaborator
|
Hi @dnhatn, I've created a changelog YAML for you. |
Collaborator
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
elasticsearchmachine
added
the
Team:Analytics
nik9000
approved these changes
Aug 15, 2025
Member
Author
|
@nik9000 Thanks so much for the feedback + review! |
javanna
pushed a commit
to javanna/elasticsearch
that referenced
this pull request
Aug 18, 2025
…lastic#132774) We have seen CPU underutilization in metrics queries against large indices when using either SEGMENT or DOC partitioning: 1. SEGMENT partitioning does not split large segments, so a single driver may process the entire query if most matching documents in a few segments. 2. DOC partitioning creates a fixed number of slices. If matching documents are concentrated in a few slices, a single driver may execute the entire query. This PR introduces dynamic-sized partitioning for DOC to address CPU underutilization while keeping overhead small: Partitioning starts with a desired partition size based on task_concurrency and caps the slice size at approximately 250K documents, preventing underutilization when matching documents are concentrated in one area. For small and medium segments (less than five times the desired slice size), a variant of segment partitioning is used, which also splits segments larger than the desired size as needed. To prevent multiple drivers from working on the same large segment unnecessarily, a single driver processes a segment sequentially until work-stealing occurs. This is accomplished by passing the current slice when polling for the next, allowing the queue to provide the next sequential slice from the same segment. New drivers are assigned slices from segments not currently being processed.
javanna
pushed a commit
to javanna/elasticsearch
that referenced
this pull request
Aug 18, 2025
…33038) With elastic#132774, the overhead of running queries with DOC partitioning is small. While we might switch the default data partitioning to DOC for all queries in the future, this PR defaults data partitioning to DOC for time-series queries only to minimize any unexpected impact. Relates elastic#132774
rjernst
pushed a commit
to rjernst/elasticsearch
that referenced
this pull request
Aug 18, 2025
…33038) With elastic#132774, the overhead of running queries with DOC partitioning is small. While we might switch the default data partitioning to DOC for all queries in the future, this PR defaults data partitioning to DOC for time-series queries only to minimize any unexpected impact. Relates elastic#132774
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have seen CPU underutilization in metrics queries against large indices when using either SEGMENT or DOC partitioning:
This PR introduces dynamic-sized partitioning for DOC to address CPU underutilization while keeping overhead small:
Partitioning starts with a desired partition size based on task_concurrency and caps the slice size at approximately 500K documents, preventing underutilization when matching documents are concentrated in one area.
For small and medium segments (less than five times the desired slice size), a variant of segment partitioning is used, which also splits segments larger than the desired size as needed.
To prevent multiple drivers from working on the same large segment unnecessarily, a single driver processes a segment sequentially until work-stealing occurs. This is accomplished by passing the current slice when polling for the next, allowing the queue to provide the next sequential slice from the same segment. New drivers are assigned slices from segments not currently being processed.