Skip to content

Commit f76826c

Browse files
ajleong623ajleong623
authored
Adding Cluster Endpoints for search failures and refactoring document status counters (#19115)
* add first changes Signed-off-by: Anthony Leong <[email protected]> * test for breaking detection Signed-off-by: Anthony Leong <[email protected]> * test for breaking detection Signed-off-by: Anthony Leong <[email protected]> * add changes Signed-off-by: Anthony Leong <[email protected]> * fix compile issues Signed-off-by: Anthony Leong <[email protected]> * fix breaking changes Signed-off-by: Anthony Leong <[email protected]> * fix breaking changes Signed-off-by: Anthony Leong <[email protected]> * added integration tests Signed-off-by: Anthony Leong <[email protected]> * fix null pointer and breaking changes Signed-off-by: Anthony Leong <[email protected]> * test runner not found Signed-off-by: Anthony Leong <[email protected]> * test compilation Signed-off-by: Anthony Leong <[email protected]> * spotless Signed-off-by: Anthony Leong <[email protected]> * spotless Signed-off-by: Anthony Leong <[email protected]> * it should pass Signed-off-by: Anthony Leong <[email protected]> * refactor doc status Signed-off-by: Anthony Leong <[email protected]> * refactor doc status Signed-off-by: Anthony Leong <[email protected]> * refactor doc status Signed-off-by: Anthony Leong <[email protected]> * rerun Signed-off-by: Anthony Leong <[email protected]> * skip old versions without status counter Signed-off-by: Anthony Leong <[email protected]> * updated license Signed-off-by: Anthony Leong <[email protected]> * retry for flaky Signed-off-by: Anthony Leong <[email protected]> * retry for flaky Signed-off-by: Anthony Leong <[email protected]> * retry for flaky Signed-off-by: Anthony Leong <[email protected]> * retry for flaky Signed-off-by: Anthony Leong <[email protected]> * keystore test timed out Signed-off-by: Anthony Leong <[email protected]> * keystore test timed out Signed-off-by: Anthony Leong <[email protected]> * refactor enum and stat holders Signed-off-by: Anthony Leong <[email protected]> * applied suggestions Signed-off-by: Anthony Leong <[email protected]> * retry for flaky Signed-off-by: Anthony Leong <[email protected]> * retry for flaky Signed-off-by: Anthony Leong <[email protected]> * retry for flaky Signed-off-by: Anthony Leong <[email protected]> * update conflicts Signed-off-by: Anthony Leong <[email protected]> * revert merge scheduler config Signed-off-by: Anthony Leong <[email protected]> * remove duplicate method Signed-off-by: Anthony Leong <[email protected]> * compile Signed-off-by: Anthony Leong <[email protected]> * update versions Signed-off-by: Anthony Leong <[email protected]> * comment out doc status Signed-off-by: Anthony Leong <[email protected]> * spotless Signed-off-by: Anthony Leong <[email protected]> * spotless Signed-off-by: Anthony Leong <[email protected]> * add back old doc values Signed-off-by: Anthony Leong <[email protected]> * spotless Signed-off-by: Anthony Leong <[email protected]> * remove docstatus again Signed-off-by: Anthony Leong <[email protected]> * retry for flaky Signed-off-by: Anthony Leong <[email protected]> * did not run Signed-off-by: Anthony Leong <[email protected]> * update version rules Signed-off-by: Anthony Leong <[email protected]> * rerun for flaky Signed-off-by: Anthony Leong <[email protected]> * rerun for flaky Signed-off-by: Anthony Leong <[email protected]> * should be ready Signed-off-by: Anthony Leong <[email protected]> * spotless Signed-off-by: Anthony Leong <[email protected]> * coderabbit suggestions Signed-off-by: Anthony Leong <[email protected]> * applied suggestions Signed-off-by: Anthony Leong <[email protected]> * write booleans Signed-off-by: Anthony Leong <[email protected]> * fix typos Signed-off-by: Anthony Leong <[email protected]> * rerun Signed-off-by: Anthony Leong <[email protected]> * update skip versions Signed-off-by: Anthony Leong <[email protected]> * remove duplicate yml test Signed-off-by: Anthony Leong <[email protected]> --------- Signed-off-by: Anthony Leong <[email protected]>
1 parent 7d97f77 commit f76826c

File tree

24 files changed

+885
-82
lines changed

24 files changed

+885
-82
lines changed

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
3333
- Implement error_trace parameter for bulk requests ([#19985](https://github.com/opensearch-project/OpenSearch/pull/19985))
3434
- Allow the truncate filter in normalizers ([#19778](https://github.com/opensearch-project/OpenSearch/issues/19778))
3535
- Support pull-based ingestion message mappers and raw payload support ([#19765](https://github.com/opensearch-project/OpenSearch/pull/19765))
36+
- Add search API tracker ([#18601](https://github.com/opensearch-project/OpenSearch/pull/18601))
3637
- Support dynamic consumer configuration update in pull-based ingestion ([#19963](https://github.com/opensearch-project/OpenSearch/pull/19963))
3738

3839
### Changed

‎libs/core/src/main/java/org/opensearch/core/rest/RestStatus.java‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -536,6 +536,19 @@ public int getStatusFamilyCode() {
536536
return status / 100;
537537
}
538538

539+
public String getStatusType() {
540+
int family = getStatusFamilyCode();
541+
if (family <= 3) {
542+
return StatusType.SUCCESS.toString();
543+
} else if (family == 4) {
544+
// 4xx errors correspond with errors on the client side
545+
return StatusType.USER_ERROR.toString();
546+
} else {
547+
// 5xx errors correspond with errors on the systems side
548+
return StatusType.SYSTEM_FAILURE.toString();
549+
}
550+
}
551+
539552
public static RestStatus readFrom(StreamInput in) throws IOException {
540553
return RestStatus.valueOf(in.readString());
541554
}

‎libs/core/src/main/java/org/opensearch/core/rest/StatusType.java‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.core.rest;
10+
11+
import java.util.Locale;
12+
13+
/**
14+
* Enum for status type which keeps track of potential response families in {@link RestStatus}
15+
*
16+
* @opensearch.api
17+
*/
18+
public enum StatusType {
19+
SUCCESS,
20+
USER_ERROR,
21+
SYSTEM_FAILURE;
22+
23+
@Override
24+
public String toString() {
25+
return name().toLowerCase(Locale.ROOT);
26+
}
27+
}

‎rest-api-spec/src/main/resources/rest-api-spec/test/nodes.stats/11_indices_metrics.yml‎

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
---
22
"Metric - blank":
33
- skip:
4+
version: " - 3.3.99"
45
features: [arbitrary_key]
6+
reason: "Status Counter was introduced in v3.4.0"
57
- do:
68
nodes.info: {}
79
- set:
@@ -25,11 +27,14 @@
2527
- is_true: nodes.$node_id.indices.segments
2628
- is_true: nodes.$node_id.indices.translog
2729
- is_true: nodes.$node_id.indices.recovery
30+
- is_true: nodes.$node_id.indices.status_counter
2831

2932
---
3033
"Metric - _all":
3134
- skip:
3235
features: [arbitrary_key]
36+
version: " - 3.3.99"
37+
reason: "Status Counter was introduced in v3.4.0"
3338
- do:
3439
nodes.info: {}
3540
- set:
@@ -53,11 +58,14 @@
5358
- is_true: nodes.$node_id.indices.segments
5459
- is_true: nodes.$node_id.indices.translog
5560
- is_true: nodes.$node_id.indices.recovery
61+
- is_true: nodes.$node_id.indices.status_counter
5662

5763
---
5864
"Metric - indices _all":
5965
- skip:
6066
features: [arbitrary_key]
67+
version: " - 3.3.99"
68+
reason: "Status Counter was introduced in v3.4.0"
6169
- do:
6270
nodes.info: {}
6371
- set:
@@ -81,11 +89,14 @@
8189
- is_true: nodes.$node_id.indices.segments
8290
- is_true: nodes.$node_id.indices.translog
8391
- is_true: nodes.$node_id.indices.recovery
92+
- is_true: nodes.$node_id.indices.status_counter
8493

8594
---
8695
"Metric - one":
8796
- skip:
8897
features: [arbitrary_key]
98+
version: " - 3.3.99"
99+
reason: "Status Counter was introduced in v3.4.0"
89100
- do:
90101
nodes.info: {}
91102
- set:
@@ -109,11 +120,14 @@
109120
- is_false: nodes.$node_id.indices.segments
110121
- is_false: nodes.$node_id.indices.translog
111122
- is_false: nodes.$node_id.indices.recovery
123+
- is_true: nodes.$node_id.indices.status_counter
112124

113125
---
114126
"Metric - multi":
115127
- skip:
116128
features: [arbitrary_key]
129+
version: " - 3.3.99"
130+
reason: "Status Counter was introduced in v3.4.0"
117131
- do:
118132
nodes.info: {}
119133
- set:
@@ -137,12 +151,13 @@
137151
- is_false: nodes.$node_id.indices.segments
138152
- is_false: nodes.$node_id.indices.translog
139153
- is_false: nodes.$node_id.indices.recovery
154+
- is_true: nodes.$node_id.indices.status_counter
140155

141156
---
142157
"Metric - indexing doc_status":
143158
- skip:
144-
version: " - 2.10.99"
145-
reason: "Doc Status Stats were introduced in v2.11.0"
159+
version: " - 3.3.99"
160+
reason: "Doc Status Stats were introduced in v2.11.0 and moved to Status Counter in v3.4.0"
146161
- do:
147162
nodes.info: {}
148163
- set:
@@ -154,7 +169,6 @@
154169
- is_false: nodes.$node_id.indices.docs
155170
- is_false: nodes.$node_id.indices.store
156171
- is_true: nodes.$node_id.indices.indexing
157-
- is_true: nodes.$node_id.indices.indexing.doc_status
158172
- is_false: nodes.$node_id.indices.get
159173
- is_false: nodes.$node_id.indices.search
160174
- is_false: nodes.$node_id.indices.merges
@@ -167,6 +181,9 @@
167181
- is_false: nodes.$node_id.indices.segments
168182
- is_false: nodes.$node_id.indices.translog
169183
- is_false: nodes.$node_id.indices.recovery
184+
- is_true: nodes.$node_id.indices.status_counter
185+
- is_true: nodes.$node_id.indices.status_counter.doc_status
186+
- is_true: nodes.$node_id.indices.status_counter.search_response_status
170187

171188
---
172189
"Metric - recovery":

‎server/src/internalClusterTest/java/org/opensearch/nodestats/NodeStatsIT.java‎

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
import org.opensearch.action.admin.cluster.node.stats.NodeStats;
1414
import org.opensearch.action.admin.cluster.node.stats.NodesStatsResponse;
1515
import org.opensearch.action.admin.indices.stats.CommonStatsFlags;
16+
import org.opensearch.action.admin.indices.stats.DocStatusStats;
1617
import org.opensearch.action.bulk.BulkItemResponse;
1718
import org.opensearch.action.bulk.BulkRequest;
1819
import org.opensearch.action.bulk.BulkResponse;
@@ -32,7 +33,6 @@
3233
import org.opensearch.index.IndexNotFoundException;
3334
import org.opensearch.index.engine.DocumentMissingException;
3435
import org.opensearch.index.engine.VersionConflictEngineException;
35-
import org.opensearch.index.shard.IndexingStats.Stats.DocStatusStats;
3636
import org.opensearch.indices.NodeIndicesStats;
3737
import org.opensearch.test.OpenSearchIntegTestCase;
3838
import org.opensearch.test.OpenSearchIntegTestCase.ClusterScope;
@@ -47,7 +47,7 @@
4747
import java.util.LinkedHashMap;
4848
import java.util.List;
4949
import java.util.Map;
50-
import java.util.concurrent.atomic.AtomicLong;
50+
import java.util.concurrent.atomic.LongAdder;
5151

5252
import static java.util.Collections.singletonMap;
5353
import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertAcked;
@@ -281,15 +281,14 @@ public void testNodeIndicesStatsDocStatsWithAggregations() {
281281
.getNodes()
282282
.get(0)
283283
.getIndices()
284-
.getIndexing()
285-
.getTotal()
284+
.getStatusCounterStats()
286285
.getDocStatusStats();
287286

288287
assertTrue(
289288
Arrays.equals(
290289
docStatusStats.getDocStatusCounter(),
291290
expectedDocStatusStats.getDocStatusCounter(),
292-
Comparator.comparingLong(AtomicLong::longValue)
291+
Comparator.comparingLong(LongAdder::longValue)
293292
)
294293
);
295294
}
@@ -543,15 +542,14 @@ private void assertDocStatusStats() {
543542
.getNodes()
544543
.get(0)
545544
.getIndices()
546-
.getIndexing()
547-
.getTotal()
545+
.getStatusCounterStats()
548546
.getDocStatusStats();
549547

550548
assertTrue(
551549
Arrays.equals(
552550
docStatusStats.getDocStatusCounter(),
553551
expectedDocStatusStats.getDocStatusCounter(),
554-
Comparator.comparingLong(AtomicLong::longValue)
552+
Comparator.comparingLong(LongAdder::longValue)
555553
)
556554
);
557555
}

‎server/src/internalClusterTest/java/org/opensearch/search/msearch/MultiSearchStatsIT.java‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.search.msearch;
10+
11+
import org.opensearch.ExceptionsHelper;
12+
import org.opensearch.action.admin.indices.stats.SearchResponseStatusStats;
13+
import org.opensearch.action.search.MultiSearchRequestBuilder;
14+
import org.opensearch.action.search.MultiSearchResponse;
15+
import org.opensearch.action.search.SearchResponse;
16+
import org.opensearch.index.query.QueryBuilders;
17+
import org.opensearch.test.OpenSearchIntegTestCase;
18+
import org.opensearch.test.OpenSearchIntegTestCase.ClusterScope;
19+
import org.opensearch.test.OpenSearchIntegTestCase.Scope;
20+
21+
import java.util.Arrays;
22+
import java.util.Comparator;
23+
import java.util.concurrent.atomic.LongAdder;
24+
25+
@ClusterScope(scope = Scope.TEST, numDataNodes = 1, numClientNodes = 0, supportsDedicatedMasters = false)
26+
public class MultiSearchStatsIT extends OpenSearchIntegTestCase {
27+
private final SearchResponseStatusStats expectedSearchResponseStatusStats = new SearchResponseStatusStats();
28+
29+
public void testNodeIndicesStatsSearchResponseStatusStatsMultiSearch() {
30+
createIndex("test");
31+
ensureGreen();
32+
client().prepareIndex("test").setId("1").setSource("field", "xxx").get();
33+
client().prepareIndex("test").setId("2").setSource("field", "yyy").get();
34+
refresh();
35+
36+
int failureCount = randomIntBetween(0, 50);
37+
int successCount = randomIntBetween(0, 50);
38+
39+
MultiSearchRequestBuilder multiSearchRequestBuilder = client().prepareMultiSearch();
40+
41+
for (int i = 0; i < failureCount; i++) {
42+
multiSearchRequestBuilder.add(client().prepareSearch("noIndex").setQuery(QueryBuilders.termQuery("field", "yyy")));
43+
}
44+
45+
for (int i = 0; i < successCount; i++) {
46+
multiSearchRequestBuilder.add(client().prepareSearch("test").setQuery(QueryBuilders.matchAllQuery()));
47+
}
48+
49+
MultiSearchResponse response = multiSearchRequestBuilder.get();
50+
51+
for (MultiSearchResponse.Item item : response) {
52+
if (item.isFailure()) {
53+
updateExpectedDocStatusCounter(expectedSearchResponseStatusStats, item.getFailure());
54+
} else {
55+
updateExpectedDocStatusCounter(expectedSearchResponseStatusStats, item.getResponse());
56+
}
57+
}
58+
assertSearchResponseStatusStats();
59+
}
60+
61+
private void assertSearchResponseStatusStats() {
62+
SearchResponseStatusStats searchResponseStatusStats = client().admin()
63+
.cluster()
64+
.prepareNodesStats()
65+
.execute()
66+
.actionGet()
67+
.getNodes()
68+
.get(0)
69+
.getIndices()
70+
.getStatusCounterStats()
71+
.getSearchResponseStatusStats();
72+
73+
assertTrue(
74+
Arrays.equals(
75+
searchResponseStatusStats.getSearchResponseStatusCounter(),
76+
expectedSearchResponseStatusStats.getSearchResponseStatusCounter(),
77+
Comparator.comparingLong(LongAdder::longValue)
78+
)
79+
);
80+
}
81+
82+
private void updateExpectedDocStatusCounter(SearchResponseStatusStats expectedSearchResponseStatusStats, SearchResponse r) {
83+
expectedSearchResponseStatusStats.inc(r.status());
84+
}
85+
86+
private void updateExpectedDocStatusCounter(SearchResponseStatusStats expectedSearchResponseStatusStats, Exception e) {
87+
expectedSearchResponseStatusStats.inc(ExceptionsHelper.status(e));
88+
}
89+
}

0 commit comments

Comments
 (0)