Improve accuracy of write load forecast when shard numbers change #129990
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nicktindall
added
>bug
:Distributed Coordination/Allocation
elasticsearchmachine
added
the
Team:Distributed Coordination
Collaborator
|
Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination) |
nicktindall
commented
Jun 25, 2025
| * If the auto sharding service thinks the number of shards must be changed but it can't recommend a change due to the cooldown | ||
| * period not lapsing, the result will be of type {@link AutoShardingType#COOLDOWN_PREVENTED_INCREASE} or | ||
| * {@link AutoShardingType#COOLDOWN_PREVENTED_INCREASE} with the remaining cooldown configured and the number of shards that should | ||
| * {@link AutoShardingType#COOLDOWN_PREVENTED_DECREASE} with the remaining cooldown configured and the number of shards that should |
Contributor
Author
There was a problem hiding this comment.
Fixed a typo too
Collaborator
|
Hi @nicktindall, I've created a changelog YAML for you. |
mhl-b
reviewed
Jun 25, 2025
nicktindall
commented
Jun 26, 2025
| closeTo( | ||
| originalWriteIndexMetadata.getNumberOfShards() * writeLoadForecaster.getForecastedWriteLoad(originalWriteIndexMetadata) | ||
| .getAsDouble(), | ||
| 0.01 |
Contributor
Author
There was a problem hiding this comment.
There are sometimes small rounding errors
nicktindall
changed the title
mridula-s109
pushed a commit
to mridula-s109/elasticsearch
that referenced
this pull request
Jul 3, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When the number of shards changes in a data-stream, whether due to auto-sharding or manual intervention, it impacts the write-load forecast used for the new write index. This is because the forecast is calculated as a weighted average per-shard write load. So if the number of shards increased in the new write index, the indexes total write load will increase correspondingly, likewise if the number of shards decreased. This PR shifts to calculating a per-index write load, so it can be correctly adjusted when the shard count changes.
Explanation
Write load is calculated per shard, it is equal to
WRITE_LOAD(shard) = WRITE_TIME(shard) / UPTIME(shard)To calculate the forecasted write load of the shards of a new index on DS rollover, we calculate the weighted average write load of all the shards of all the "recent" indices, e.g.
FORECAST_WRITE_LOAD(shard) = sum(WRITE_LOAD(shard)) / sum(UPTIME(shard)) | all shard in RECENT_INDEXWe use the forecasted write load quite heavily in balancing.
To calculate the write load for a node, we add the write loads of all the shards in all the allocated indices
FORECAST_WRITE_LOAD(node) = sum(FORECAST_WRITE_LOAD(shard)) | all shards on node(see
org.elasticsearch.cluster.routing.allocation.allocator.BalancedShardsAllocator.ModelNode#(addShard|removeShard))And to calculate the write load for an index, we add the write loads of all the shards in the index
FORECAST_WRITE_LOAD(index) = sum(FORECAST_WRITE_LOAD(shard)) | all shards in index (taking into account replicas)(see
org.elasticsearch.cluster.routing.allocation.allocator.WeightFunction#getIndexWriteLoad)This means when auto-sharding decides to increase the number of shards, the total write load in the cluster increases, because
e.g. an index that went from 3 shards to 6 shards goes from
3 * FORECAST_WRITE_LOAD(shard)to6 * FORECAST_WRITE_LOAD(shard)Conversely when auto-sharding decides to decrease the number of shards, the total write load in the cluster will decrease
Why is this a problem?
If we auto-shard down from 63 to 38 shards in a rollover, like we did the scale testing environment, the balancer will use the forecasted write-load that was calculated for 63 shards, but with only 38 shards. That load from 63 shards will be compressed into the 38 shards that replace them. So the balancer will under-estimate the write load for each shard by
(1 - (38/36)) = ~40%. This will make it think it's in a desired state, but will leave nodes that have a greater proportion of the scaled-down (hence under-estimated) index's shards overloaded.