Skip to content

Unthrottle small merges #126914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Unthrottle small merges #126914

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions server/src/main/java/org/elasticsearch/index/engine/ThreadPoolMergeScheduler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,14 @@ public class ThreadPoolMergeScheduler extends MergeScheduler implements Elastics
true,
Setting.Property.NodeScope
);
/**
* Placeholder value used when max segments count of merge tasks is unspecified (which is the case of regular, non-forced, merges)
*/
private static final int UNBOUNDED_MAX_MERGE_SEGMENTS = -1;
/**
* Merges below this size are never IO throttled.
*/
private static final ByteSizeValue MIN_BIG_MERGE_SIZE = ByteSizeValue.ofMb(50L);
private final ShardId shardId;
private final MergeSchedulerConfig config;
protected final Logger logger;
Expand Down Expand Up @@ -173,8 +181,10 @@ boolean submitNewMergeTask(MergeSource mergeSource, MergePolicy.OneMerge merge,

// package-private for tests
MergeTask newMergeTask(MergeSource mergeSource, MergePolicy.OneMerge merge, MergeTrigger mergeTrigger) {
// forced merges, as well as merges triggered when closing a shard, always run un-IO-throttled
boolean isAutoThrottle = mergeTrigger != MergeTrigger.CLOSING && merge.getStoreMergeInfo().mergeMaxNumSegments() == -1;
// forced merges, small merges, as well as merges triggered when closing a shard always run un-IO-throttled
boolean isAutoThrottle = mergeTrigger != MergeTrigger.CLOSING
&& merge.getStoreMergeInfo().estimatedMergeBytes() > MIN_BIG_MERGE_SIZE.getBytes()
&& merge.getStoreMergeInfo().mergeMaxNumSegments() == UNBOUNDED_MAX_MERGE_SEGMENTS;
// IO throttling cannot be toggled for existing merge tasks, only new merge tasks pick up the updated IO throttling setting
return new MergeTask(
mergeSource,
Expand Down