[AUTOCUT] Gradle Check Flaky Test Report for ShardsLimitAllocationDeciderIT

[opensearch-ci-bot]

Flaky Test Report for ShardsLimitAllocationDeciderIT

Noticed the ShardsLimitAllocationDeciderIT has some flaky, failing tests that failed during post-merge actions and timer trigerred run on main branch.

Details

Git Reference	Merged Pull Request	Build Details	Test Name
main	null	66227 66316 66350 66422 66457 66616 66804 66915 66952 66989 67020 67043 67326 67474 67553 67608 67851 67916 67974 68220 68361 68379 68464 68569 68699 68747 68759 68839 68974 69077 69309 69324 69355 69360 69462 69583 69708 69761 69935 69936 70065 70067 70103 70130 70224 70253 70281 70491 70639 70796 70846 71078 71099 71291 71339 71414 71517 71567 71657 71665 71903 72056 72122 72143 72152 72308 72500	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
0e2783c	20842	72436	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
c0fc9ae	20679	71617	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
839254d	20532	70926	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
4fadec8	20528	70851	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
4eb4e99	20493	70671	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
5041d43	20361	70389 70404	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
d969da9	20348	69527	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
fc01431	20240	68982	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
cdc40c5	20010	67344	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
f229797	19835	66662	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
4a07236	19253	66475	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
1173bdf	19677	65924	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
84cc00c	19619	65563	org.opensearch.cluster.routing.allocation.decider.ShardsLimitAllocationDeciderIT.testCombinedClusterAndIndexSpecificShardLimits {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the ShardsLimitAllocationDeciderIT class are:

• 19958
• 20017
• 20503
• 20742
• 18523
• 19644
• 19746
• 19798
• 19966
• 20075
• 20258
• 20432
• 20434
• 20463
• 20512
• 20549
• 20824
• 18491
• 18890
• 19178
• 19248
• 19442
• 19573
• 19630
• 19631
• 19652
• 19688
• 19704
• 19734
• 19737
• 19765
• 19766
• 19788
• 19857
• 19913
• 19936
• 19947
• 19948
• 19961
• 20015
• 20023
• 20042
• 20055
• 20065
• 20067
• 20095
• 20102
• 20112
• 20115
• 20143
• 20145
• 20160
• 20184
• 20192
• 20205
• 20250
• 20265
• 20281
• 20284
• 20343
• 20346
• 20350
• 20358
• 20359
• 20368
• 20372
• 20383
• 20393
• 20422
• 20442
• 20467
• 20477
• 20480
• 20511
• 20526
• 20530
• 20538
• 20542
• 20566
• 20584
• 20589
• 20606
• 20641
• 20648
• 20650
• 20678
• 20682
• 20702
• 20713
• 20726
• 20760
• 20775
• 20792
• 20806
• 20823
• 20833
• 20840
• 20841
• 20857

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.

[andrross]

Catch All Triage - 1 2

[andrross]

I believe the issue here is that RemoteShardsBalancer is rather limited compared to LocalShardsBalancer and the near perfect bin-packing required by the testCombinedClusterAndIndexSpecificShardLimits test sometimes causes RemoteShardsBalancer to get to an end state where it leaves a shard unassigned. Here are the detailed findings from kiro:

The RemoteShardsBalancer (used for warm/remote-capable shards) has a fundamentally different and simpler allocation strategy than the LocalShardsBalancer:

1. Shuffled node order — getShuffledRemoteNodes() randomizes which node gets tried first. This means allocation order varies per run.

2. Greedy, non-backtracking allocation — It iterates shards index-by-index and greedily assigns each to the next available node. When a node can't accept a shard (e.g., cluster limit reached), it's removed from the queue. There's no weight-based optimization or sophisticated rebalancing of total shards.

3. Balance only considers primaries — The balance() method only rebalances based on primary shard count, not total shard count. So if the greedy allocation leaves total shards distributed unevenly (e.g., 6, 6, 5), the balancer won't move replicas around to free up slots.

4. No multi-step rebalance for replicas — When the greedy allocation fills two nodes to the cluster limit of 6 before placing all shards, and the remaining unplaceable shard is a replica whose primary is on the only node with capacity, the RemoteShardsBalancer won't perform the multi-step move needed to resolve this (move an existing shard off a full node to create space on a different node).

In contrast, the LocalShardsBalancer uses weight-based balancing that considers total shard count and can perform the necessary rebalancing moves to converge to the optimal 17-shard state. That's why the non-warm variant never fails — the LocalShardsBalancer handles this scenario correctly.

The warm variant fails intermittently because the shuffled node order sometimes leads to a greedy allocation that paints itself into a corner, and the RemoteShardsBalancer's limited rebalancing can't fix it within a single allocation round.