-
Notifications
You must be signed in to change notification settings - Fork 2.5k
Intra segment support for single-value metric aggregations #20503
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
prudhvigodithi
merged 33 commits into
opensearch-project:main
from
prudhvigodithi:approx
Mar 13, 2026
Merged
Changes from 26 commits
Commits
Show all changes
33 commits
Select commit
Hold shift + click to select a range
c749beb
intra segment support for query_string
prudhvigodithi 0f62870
update changelog
prudhvigodithi 0d5eb3c
Intra segment support for single-value metric aggregations
prudhvigodithi c8a7992
upstream fetch
prudhvigodithi 42be1ed
update CHANGELOG.md
prudhvigodithi d654cc2
Upstream fetch
prudhvigodithi f0db14e
Add tests
prudhvigodithi aafbc28
Add tests
prudhvigodithi 57af42e
Fix tests
prudhvigodithi 1ea8af5
Temp revert intra for cardinality agg
prudhvigodithi 7187fc3
Enable intra for cardinality agg
prudhvigodithi 0e823af
test: Add unit tests for intra-segment search support
prudhvigodithi b6954da
Fix commit issue
prudhvigodithi 398a5a3
Address github comments
prudhvigodithi 32c81a6
Address github comments
prudhvigodithi bd366f3
Add wipeIndices to tests
prudhvigodithi e047a11
Merge remote-tracking branch 'upstream/main' into approx
prudhvigodithi 5b63598
Update the IT tests
prudhvigodithi 3eb0864
Update the IT tests
prudhvigodithi b14376d
Merge remote-tracking branch 'upstream/main' into approx
prudhvigodithi 8b306ac
Fetch upstream
prudhvigodithi 26d8a98
Edge case bug with indexBulkWithSegments
prudhvigodithi 0ba1942
Address comments
prudhvigodithi 9d380ad
Address comments
prudhvigodithi 8767987
Merge remote-tracking branch 'upstream/main' into approx
prudhvigodithi e441c2a
Upstream fetch, fix conflicts
prudhvigodithi 31440a7
Upstream fetch, fix conflicts
prudhvigodithi 874fb79
Merge remote-tracking branch 'upstream/main' into approx
prudhvigodithi 8923433
Merge remote-tracking branch 'upstream/main' into approx
prudhvigodithi bbe5ed1
Merge remote-tracking branch 'upstream/main' into approx
prudhvigodithi 526f41f
Upstream fetch
prudhvigodithi fa13dd6
Merge remote-tracking branch 'upstream/main' into approx
prudhvigodithi 5a88b05
Upstream fetch, fix conflicts
prudhvigodithi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
72 changes: 72 additions & 0 deletions
72
server/src/internalClusterTest/java/org/opensearch/search/aggregations/metrics/AvgIT.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| /* | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| * | ||
| * The OpenSearch Contributors require contributions made to | ||
| * this file be licensed under the Apache-2.0 license or a | ||
| * compatible open source license. | ||
| */ | ||
|
|
||
| package org.opensearch.search.aggregations.metrics; | ||
|
|
||
| import com.carrotsearch.randomizedtesting.annotations.ParametersFactory; | ||
|
|
||
| import org.opensearch.action.index.IndexRequestBuilder; | ||
| import org.opensearch.action.search.SearchResponse; | ||
| import org.opensearch.common.settings.Settings; | ||
| import org.opensearch.test.ParameterizedStaticSettingsOpenSearchIntegTestCase; | ||
|
|
||
| import java.util.ArrayList; | ||
| import java.util.Arrays; | ||
| import java.util.Collection; | ||
| import java.util.List; | ||
|
|
||
| import static org.opensearch.search.SearchService.CLUSTER_CONCURRENT_SEGMENT_SEARCH_SETTING; | ||
| import static org.opensearch.search.SearchService.CONCURRENT_SEGMENT_SEARCH_PARTITION_MIN_SEGMENT_SIZE; | ||
| import static org.opensearch.search.SearchService.CONCURRENT_SEGMENT_SEARCH_PARTITION_STRATEGY; | ||
| import static org.opensearch.search.aggregations.AggregationBuilders.avg; | ||
| import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertSearchResponse; | ||
| import static org.hamcrest.Matchers.closeTo; | ||
| import static org.hamcrest.Matchers.notNullValue; | ||
|
|
||
| /** | ||
| * Integration tests for avg aggregation with concurrent segment search partition strategies. | ||
| */ | ||
| public class AvgIT extends ParameterizedStaticSettingsOpenSearchIntegTestCase { | ||
prudhvigodithi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| public AvgIT(Settings staticSettings) { | ||
| super(staticSettings); | ||
| } | ||
|
|
||
| @ParametersFactory | ||
| public static Collection<Object[]> parameters() { | ||
| return Arrays.asList( | ||
| new Object[] { Settings.builder().put(CLUSTER_CONCURRENT_SEGMENT_SEARCH_SETTING.getKey(), false).build() }, | ||
| new Object[] { Settings.builder().put(CONCURRENT_SEGMENT_SEARCH_PARTITION_STRATEGY.getKey(), "segment").build() }, | ||
| new Object[] { Settings.builder().put(CONCURRENT_SEGMENT_SEARCH_PARTITION_STRATEGY.getKey(), "force").build() }, | ||
| new Object[] { | ||
| Settings.builder() | ||
| .put(CONCURRENT_SEGMENT_SEARCH_PARTITION_STRATEGY.getKey(), "balanced") | ||
| .put(CONCURRENT_SEGMENT_SEARCH_PARTITION_MIN_SEGMENT_SIZE.getKey(), 1000) | ||
| .build() } | ||
| ); | ||
| } | ||
|
|
||
| public void testAvgAggregation() throws Exception { | ||
| createIndex("test_avg_agg", Settings.builder().put("index.number_of_shards", 2).put("index.number_of_replicas", 1).build()); | ||
| try { | ||
| List<IndexRequestBuilder> builders = new ArrayList<>(5000); | ||
| for (int i = 0; i < 5000; i++) { | ||
| builders.add(client().prepareIndex("test_avg_agg").setSource("value", i + 1)); | ||
| } | ||
| indexBulkWithSegments(builders, 2); | ||
| indexRandomForConcurrentSearch("test_avg_agg"); | ||
| SearchResponse response = client().prepareSearch("test_avg_agg").addAggregation(avg("avg_agg").field("value")).get(); | ||
| assertSearchResponse(response); | ||
| Avg avgAgg = response.getAggregations().get("avg_agg"); | ||
| assertThat(avgAgg, notNullValue()); | ||
| assertThat(avgAgg.getValue(), closeTo(2500.5, 0.1)); | ||
| } finally { | ||
| internalCluster().wipeIndices("test_avg_agg"); | ||
| } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
72 changes: 72 additions & 0 deletions
72
server/src/internalClusterTest/java/org/opensearch/search/aggregations/metrics/MaxIT.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| /* | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| * | ||
| * The OpenSearch Contributors require contributions made to | ||
| * this file be licensed under the Apache-2.0 license or a | ||
| * compatible open source license. | ||
| */ | ||
|
|
||
| package org.opensearch.search.aggregations.metrics; | ||
|
|
||
| import com.carrotsearch.randomizedtesting.annotations.ParametersFactory; | ||
|
|
||
| import org.opensearch.action.index.IndexRequestBuilder; | ||
| import org.opensearch.action.search.SearchResponse; | ||
| import org.opensearch.common.settings.Settings; | ||
| import org.opensearch.test.ParameterizedStaticSettingsOpenSearchIntegTestCase; | ||
|
|
||
| import java.util.ArrayList; | ||
| import java.util.Arrays; | ||
| import java.util.Collection; | ||
| import java.util.List; | ||
|
|
||
| import static org.opensearch.search.SearchService.CLUSTER_CONCURRENT_SEGMENT_SEARCH_SETTING; | ||
| import static org.opensearch.search.SearchService.CONCURRENT_SEGMENT_SEARCH_PARTITION_MIN_SEGMENT_SIZE; | ||
| import static org.opensearch.search.SearchService.CONCURRENT_SEGMENT_SEARCH_PARTITION_STRATEGY; | ||
| import static org.opensearch.search.aggregations.AggregationBuilders.max; | ||
| import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertSearchResponse; | ||
| import static org.hamcrest.Matchers.closeTo; | ||
| import static org.hamcrest.Matchers.notNullValue; | ||
|
|
||
| /** | ||
| * Integration tests for max aggregation with concurrent segment search partition strategies. | ||
| */ | ||
| public class MaxIT extends ParameterizedStaticSettingsOpenSearchIntegTestCase { | ||
|
|
||
| public MaxIT(Settings staticSettings) { | ||
| super(staticSettings); | ||
| } | ||
|
|
||
| @ParametersFactory | ||
| public static Collection<Object[]> parameters() { | ||
| return Arrays.asList( | ||
| new Object[] { Settings.builder().put(CLUSTER_CONCURRENT_SEGMENT_SEARCH_SETTING.getKey(), false).build() }, | ||
| new Object[] { Settings.builder().put(CONCURRENT_SEGMENT_SEARCH_PARTITION_STRATEGY.getKey(), "segment").build() }, | ||
| new Object[] { Settings.builder().put(CONCURRENT_SEGMENT_SEARCH_PARTITION_STRATEGY.getKey(), "force").build() }, | ||
| new Object[] { | ||
| Settings.builder() | ||
| .put(CONCURRENT_SEGMENT_SEARCH_PARTITION_STRATEGY.getKey(), "balanced") | ||
| .put(CONCURRENT_SEGMENT_SEARCH_PARTITION_MIN_SEGMENT_SIZE.getKey(), 1000) | ||
| .build() } | ||
| ); | ||
| } | ||
|
|
||
| public void testMaxAggregation() throws Exception { | ||
| createIndex("test_max_agg", Settings.builder().put("index.number_of_shards", 2).put("index.number_of_replicas", 1).build()); | ||
| try { | ||
| List<IndexRequestBuilder> builders = new ArrayList<>(5000); | ||
| for (int i = 0; i < 5000; i++) { | ||
| builders.add(client().prepareIndex("test_max_agg").setSource("value", i + 1)); | ||
| } | ||
| indexBulkWithSegments(builders, 2); | ||
| indexRandomForConcurrentSearch("test_max_agg"); | ||
| SearchResponse response = client().prepareSearch("test_max_agg").addAggregation(max("max_agg").field("value")).get(); | ||
| assertSearchResponse(response); | ||
| Max maxAgg = response.getAggregations().get("max_agg"); | ||
| assertThat(maxAgg, notNullValue()); | ||
| assertThat(maxAgg.getValue(), closeTo(5000.0, 0.1)); | ||
| } finally { | ||
| internalCluster().wipeIndices("test_max_agg"); | ||
| } | ||
| } | ||
| } |
72 changes: 72 additions & 0 deletions
72
server/src/internalClusterTest/java/org/opensearch/search/aggregations/metrics/MinIT.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| /* | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| * | ||
| * The OpenSearch Contributors require contributions made to | ||
| * this file be licensed under the Apache-2.0 license or a | ||
| * compatible open source license. | ||
| */ | ||
|
|
||
| package org.opensearch.search.aggregations.metrics; | ||
|
|
||
| import com.carrotsearch.randomizedtesting.annotations.ParametersFactory; | ||
|
|
||
| import org.opensearch.action.index.IndexRequestBuilder; | ||
| import org.opensearch.action.search.SearchResponse; | ||
| import org.opensearch.common.settings.Settings; | ||
| import org.opensearch.test.ParameterizedStaticSettingsOpenSearchIntegTestCase; | ||
|
|
||
| import java.util.ArrayList; | ||
| import java.util.Arrays; | ||
| import java.util.Collection; | ||
| import java.util.List; | ||
|
|
||
| import static org.opensearch.search.SearchService.CLUSTER_CONCURRENT_SEGMENT_SEARCH_SETTING; | ||
| import static org.opensearch.search.SearchService.CONCURRENT_SEGMENT_SEARCH_PARTITION_MIN_SEGMENT_SIZE; | ||
| import static org.opensearch.search.SearchService.CONCURRENT_SEGMENT_SEARCH_PARTITION_STRATEGY; | ||
| import static org.opensearch.search.aggregations.AggregationBuilders.min; | ||
| import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertSearchResponse; | ||
| import static org.hamcrest.Matchers.closeTo; | ||
| import static org.hamcrest.Matchers.notNullValue; | ||
|
|
||
| /** | ||
| * Integration tests for min aggregation with concurrent segment search partition strategies. | ||
| */ | ||
| public class MinIT extends ParameterizedStaticSettingsOpenSearchIntegTestCase { | ||
|
|
||
| public MinIT(Settings staticSettings) { | ||
| super(staticSettings); | ||
| } | ||
|
|
||
| @ParametersFactory | ||
| public static Collection<Object[]> parameters() { | ||
| return Arrays.asList( | ||
| new Object[] { Settings.builder().put(CLUSTER_CONCURRENT_SEGMENT_SEARCH_SETTING.getKey(), false).build() }, | ||
| new Object[] { Settings.builder().put(CONCURRENT_SEGMENT_SEARCH_PARTITION_STRATEGY.getKey(), "segment").build() }, | ||
| new Object[] { Settings.builder().put(CONCURRENT_SEGMENT_SEARCH_PARTITION_STRATEGY.getKey(), "force").build() }, | ||
| new Object[] { | ||
| Settings.builder() | ||
| .put(CONCURRENT_SEGMENT_SEARCH_PARTITION_STRATEGY.getKey(), "balanced") | ||
| .put(CONCURRENT_SEGMENT_SEARCH_PARTITION_MIN_SEGMENT_SIZE.getKey(), 1000) | ||
| .build() } | ||
| ); | ||
| } | ||
|
|
||
| public void testMinAggregation() throws Exception { | ||
| createIndex("test_min_agg", Settings.builder().put("index.number_of_shards", 2).put("index.number_of_replicas", 1).build()); | ||
| try { | ||
| List<IndexRequestBuilder> builders = new ArrayList<>(5000); | ||
| for (int i = 0; i < 5000; i++) { | ||
| builders.add(client().prepareIndex("test_min_agg").setSource("value", i + 1)); | ||
| } | ||
| indexBulkWithSegments(builders, 2); | ||
| indexRandomForConcurrentSearch("test_min_agg"); | ||
| SearchResponse response = client().prepareSearch("test_min_agg").addAggregation(min("min_agg").field("value")).get(); | ||
| assertSearchResponse(response); | ||
| Min minAgg = response.getAggregations().get("min_agg"); | ||
| assertThat(minAgg, notNullValue()); | ||
| assertThat(minAgg.getValue(), closeTo(1.0, 0.1)); | ||
| } finally { | ||
| internalCluster().wipeIndices("test_min_agg"); | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.