Skip to content

Prevent auto-sharding for data streams in LOOKUP index mode #131429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jul 29, 2025
Merged

Prevent auto-sharding for data streams in LOOKUP index mode #131429

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d3acd1e
Prevent auto-sharding for data streams in LOOKUP index mode
lukewhiting Jul 17, 2025
d580af0
Update docs/changelog/131429.yaml
lukewhiting Jul 17, 2025
d089204
Merge branch 'main' into ES-12330-prevent-auto-shard-on-lookup-index
lukewhiting Jul 17, 2025
0a3bc83
Reduce test duplication
lukewhiting Jul 25, 2025
50805b1
Merge branch 'main' into ES-12330-prevent-auto-shard-on-lookup-index
lukewhiting Jul 25, 2025
eeeeb50
Merge branch 'main' into ES-12330-prevent-auto-shard-on-lookup-index
lukewhiting Jul 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/changelog/131429.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pr: 131429
summary: Prevent auto-sharding for data streams in LOOKUP index mode
area: Data streams
type: bug
issues: []
7 changes: 7 additions & 0 deletions ...java/org/elasticsearch/action/datastreams/autosharding/DataStreamAutoShardingService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
import org.elasticsearch.core.Nullable;
import org.elasticsearch.core.TimeValue;
import org.elasticsearch.index.Index;
import org.elasticsearch.index.IndexMode;
import org.elasticsearch.index.shard.IndexingStats;

import java.util.Arrays;
Expand Down Expand Up @@ -371,6 +372,7 @@ public String toString() {
* there'll be no new auto sharding event)
*/
public AutoShardingResult calculate(ProjectState state, DataStream dataStream, @Nullable IndexStats writeIndexStats) {

if (isAutoShardingEnabled == false) {
logger.debug("Data stream auto-sharding service is not enabled.");
return NOT_APPLICABLE_RESULT;
Expand All @@ -385,6 +387,11 @@ public AutoShardingResult calculate(ProjectState state, DataStream dataStream, @
return NOT_APPLICABLE_RESULT;
}

if (dataStream.getIndexMode() == IndexMode.LOOKUP) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change lgtm!

if you wouldn't mind me asking a few (hopefully not dumb) Qs just to learn, just if you know them off the top of your head, i can investigate anything you might not know

  1. terminology: it seems in the code and slack discussion regarding the ticket, it seems we use the term shard to only mean write shard, but we also have replica/read shards right? just wondering whether whenever i see the term shard should i basically assume this means write shard/primary
  2. so LOOKUP is a type of index, that only only ever has one primary shard, but can it have more than 1 replica/read shard? assuming it can, do we have any replica auto-sharding in place in data streams if the read load gets too heavy?
  3. why does LOOKUP only ever allow one primary shard? it's always possible it has heavy writes (hence the error for scaleup i'm assuming), is it basically just a mode where you're assuming up-front write volume is low and it's -- as in the name -- just a lookup
  4. for this auto-sharding result logic, if we scale up, do we roll-over to a new index with more primaries or split? and if rollover the new index has more primaries right
  5. for the ticket, did the lookup index just completely fail to rollover and kept accumulating data, or it just threw the validation exceptions as this kept being called to scale-up but eventually was rolled over due to size/time?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@szybia We can chat about this in our 1:1 tomorrow if you like.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes please! might take the load off luke...

logger.debug("Data stream [{}] has indexing mode LOOKUP; auto-sharding is not applicable.", dataStream.getName());
return NOT_APPLICABLE_RESULT;
}

if (writeIndexStats == null) {
logger.debug(
"Data stream auto-sharding service cannot compute the optimal number of shards for data stream [{}] as the write index "
Expand Down
42 changes: 42 additions & 0 deletions ...org/elasticsearch/action/datastreams/autosharding/DataStreamAutoShardingServiceTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1343,4 +1343,46 @@ private static Decision.Inputs createDecisionInputsForPeriodLoggerTests(int writ
3
);
}

public void testCalculateReturnsNotApplicableForLookupIndexMode() {
var projectId = randomProjectIdOrDefault();
ProjectMetadata.Builder builder = ProjectMetadata.builder(projectId);
DataStream dataStream = createLookupModeDataStream(builder);
ClusterState state = createClusterStateWithDataStream(builder);

AutoShardingResult autoShardingResult = service.calculate(
state.projectState(projectId),
dataStream,
createIndexStats(1, 1.0, 1.0, 1.0)
);
assertThat(autoShardingResult, is(NOT_APPLICABLE_RESULT));
assertThat(decisionsLogged, hasSize(0));
}

public void testCalculateReturnsNotApplicableForLookupIndexModeWithNullStats() {
var projectId = randomProjectIdOrDefault();
ProjectMetadata.Builder builder = ProjectMetadata.builder(projectId);
DataStream dataStream = createLookupModeDataStream(builder);
ClusterState state = createClusterStateWithDataStream(builder);

AutoShardingResult autoShardingResult = service.calculate(state.projectState(projectId), dataStream, null);
assertThat(autoShardingResult, is(NOT_APPLICABLE_RESULT));
assertThat(decisionsLogged, hasSize(0));
}

private DataStream createLookupModeDataStream(ProjectMetadata.Builder builder) {
DataStream dataStream = DataStream.builder(dataStreamName, List.of(new Index("test-index", randomUUID())))
.setGeneration(1)
.setIndexMode(IndexMode.LOOKUP)
.build();
builder.put(dataStream);
return dataStream;
}

private ClusterState createClusterStateWithDataStream(ProjectMetadata.Builder builder) {
return ClusterState.builder(ClusterName.DEFAULT)
.nodes(DiscoveryNodes.builder().add(DiscoveryNodeUtils.create("n1")))
.putProjectMetadata(builder.build())
.build();
}
}
Loading