Skip to content

Commit 8ecb2be

Browse files
ansjcykolchfa-awsnatebower
authored
Add live queries doc and update top queries doc (#9658)
* Add live queries doc and update top queries doc Signed-off-by: Jason Chenyang Ji <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Jason Chenyang Ji <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
1 parent d9a34ad commit 8ecb2be

File tree

4 files changed

+297
-8
lines changed

4 files changed

+297
-8
lines changed

‎_install-and-configure/configuring-opensearch/plugin-settings.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ The Notifications plugin supports the following settings. All settings in this l
8585

8686
## Query Insights plugin settings
8787

88-
For information about Query Insights plugin settings, see [Query insights settings]({{site.url}}{{site.baseurl}}/observing-your-data/query-insights/index#query-insights-settings).
88+
For information about Query Insights plugin settings, see [Query Insights features and settings]({{site.url}}{{site.baseurl}}/observing-your-data/query-insights/index#query-insights-features-and-settings).
8989

9090
## Security plugin settings
9191

‎_observing-your-data/query-insights/index.md‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,10 @@ To monitor and analyze the search queries within your OpenSearch cluster, you ca
1616

1717
Typical use cases for query insights features include the following:
1818

19-
- Identifying top queries by latency within specific time frames
20-
- Debugging slow search queries and latency spikes
19+
- Identify the slowest or most resource-intensive queries impacting your cluster.
20+
- Debug latency spikes and understand query performance patterns.
21+
- Analyze common slow query structures to find optimization opportunities.
22+
- Monitor live, in-flight queries to diagnose immediate search performance issues.
2123

2224
Query insights features are supported by the Query Insights plugin. At a high level, query insights features comprise the following components:
2325

@@ -35,13 +37,15 @@ bin/opensearch-plugin install query-insights
3537
```
3638
For information about installing plugins, see [Installing plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/).
3739

38-
## Query Insights settings
40+
## Query Insights features and settings
3941

40-
You can obtain the following information using Query Insights:
42+
Query Insights provides several ways to monitor and analyze your search queries:
4143

42-
- [Top n queries]({{site.url}}{{site.baseurl}}/observing-your-data/query-insights/top-n-queries/)
43-
- [Grouping top N queries]({{site.url}}{{site.baseurl}}/observing-your-data/query-insights/grouping-top-n-queries/)
44-
- [Query metrics]({{site.url}}{{site.baseurl}}/observing-your-data/query-insights/query-metrics/)
44+
- **[Top N queries]({{site.url}}{{site.baseurl}}/observing-your-data/query-insights/top-n-queries/)**: Identify the most resource-intensive or slowest queries over specific time frames based on various performance metrics.
45+
- **[Grouping top N queries]({{site.url}}{{site.baseurl}}/observing-your-data/query-insights/grouping-top-n-queries/)**: Discover patterns and analyze similar slow queries by grouping them based on query source structure.
46+
- **[Live queries monitoring]({{site.url}}{{site.baseurl}}/observing-your-data/query-insights/live-queries/)**: Get real-time visibility into search queries currently executing within your cluster to identify and debug queries that are currently long running or resource heavy.
47+
- **[Query insights dashboards]({{site.url}}{{site.baseurl}}/observing-your-data/query-insights/query-insights-dashboard/)**: Visualize and configure top query insights interactively in OpenSearch Dashboards.
48+
- **[Query metrics]({{site.url}}{{site.baseurl}}/observing-your-data/query-insights/query-metrics/)**: Understand the specific performance metrics per query type.
4549

4650
## Query Insights plugin health
4751

‎_observing-your-data/query-insights/live-queries.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
---
2+
layout: default
3+
title: Live queries
4+
parent: Query insights
5+
nav_order: 20
6+
---
7+
8+
# Live queries
9+
**Introduced 3.0**
10+
11+
Use the Live Queries API to retrieve currently running search queries across the cluster or on specific nodes. Monitoring live queries using Query Insights allows you to get real-time visibility into the search queries that are currently executing within your OpenSearch cluster. This is useful for identifying and debugging queries that might be running for an unexpectedly long time or consuming significant resources at the moment.
12+
13+
The API returns a list of currently executing search queries, sorted by a specified metric (defaulting to `latency`) in descending order. The response includes the details for each live query, such as the query source, search type, involved indexes, node ID, start time, latency, and resource usage (on the coordinator node) so far.
14+
15+
## Endpoints
16+
17+
```json
18+
GET /_insights/live_queries
19+
```
20+
21+
## Query parameters
22+
23+
The following table lists the available query parameters. All query parameters are optional.
24+
25+
| Parameter | Data type | Description |
26+
| :--- | :--- | :--- |
27+
| `verbose` | Boolean | Whether to include detailed query information in the output. Default is `true`. |
28+
| `nodeId` | String | A comma-separated list of node IDs used to filter the results. If omitted, queries from all nodes are returned. |
29+
| `sort` | String | The metric to sort the results by. Valid values are `latency`, `cpu`, or `memory`. Default is `latency`. |
30+
| `size` | Integer | The number of query records to return. Default is 100. |
31+
32+
## Example request
33+
34+
The following example request fetches the top 10 queries sorted by CPU usage, with verbose output disabled:
35+
36+
```json
37+
GET /_insights/live_queries?verbose=false&sort=cpu&size=10
38+
```
39+
{% include copy-curl.html %}
40+
41+
## Example response
42+
43+
```json
44+
{
45+
"live_queries" : [
46+
{
47+
"timestamp" : 1745359226777,
48+
"id" : "troGHNGUShqDj3wK_K5ZIw:512",
49+
"description" : "indices[my-index-*], search_type[QUERY_THEN_FETCH], source[{\"size\":20,\"query\":{\"term\":{\"user.id\":{\"value\":\"userId\",\"boost\":1.0}}}}]",
50+
"node_id" : "troGHNGUShqDj3wK_K5ZIw",
51+
"measurements" : {
52+
"latency" : {
53+
"number" : 13959364458,
54+
"count" : 1,
55+
"aggregationType" : "NONE"
56+
},
57+
"memory" : {
58+
"number" : 3104,
59+
"count" : 1,
60+
"aggregationType" : "NONE"
61+
},
62+
"cpu" : {
63+
"number" : 405000,
64+
"count" : 1,
65+
"aggregationType" : "NONE"
66+
}
67+
}
68+
},
69+
{
70+
"timestamp" : 1745359229158,
71+
"id" : "Y6eBnbdISPO6XaVfxCBRgg:454",
72+
"description" : "indices[my-index-*], search_type[QUERY_THEN_FETCH], source[{\"size\":20,\"query\":{\"term\":{\"user.id\":{\"value\":\"userId\",\"boost\":1.0}}}}]",
73+
"node_id" : "Y6eBnbdISPO6XaVfxCBRgg",
74+
"measurements" : {
75+
"latency" : {
76+
"number" : 11579097209,
77+
"count" : 1,
78+
"aggregationType" : "NONE"
79+
},
80+
"memory" : {
81+
"number" : 3104,
82+
"count" : 1,
83+
"aggregationType" : "NONE"
84+
},
85+
"cpu" : {
86+
"number" : 511000,
87+
"count" : 1,
88+
"aggregationType" : "NONE"
89+
}
90+
}
91+
}
92+
]
93+
}
94+
```
95+
96+
## Response fields
97+
98+
| Field | Data type | Description |
99+
| :------------------ | :-------- | :--------------------------------------------------------------------------------------------------------- |
100+
| `timestamp` | Long | The time at which the query task started, in milliseconds since the epoch. |
101+
| `id` | String | The unique identifier of the search request (the search task ID associated with the query). |
102+
| `description`| String | A description of the query, including the indexes on which it runs, search type, and query source. Only included if `verbose` is `true` (default). |
103+
| `node_id`| String | The coordinator node ID of the node on which the query task is running. |
104+
| `measurements` | Object | An object containing performance metrics gathered so far for the query. |
105+
| `measurements.LATENCY` | Object | Contains the `value` (current running time in nanoseconds) and `unit` (`nanos`). |
106+
| `measurements.CPU` | Object | Contains the `value` (CPU time consumed so far in nanoseconds) and `unit` (`nanos`). |
107+
| `measurements.MEMORY` | Object | Contains the `value` (heap memory used so far in bytes) and `unit` (`bytes`). |

‎_observing-your-data/query-insights/top-n-queries.md‎

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,184 @@ Parameter | Data type | Description
9191
`id` | String | The ID of a specific top query record to retrieve.
9292
`verbose` | Boolean | Indicates whether to return verbose output. Default is `true`.
9393

94+
### Example response
95+
96+
<details markdown="block">
97+
<summary>
98+
Response
99+
</summary>
100+
{: .text-delta}
101+
102+
```json
103+
{
104+
"top_queries" : [
105+
{
106+
"timestamp" : 1745021834451,
107+
"id" : "36506bd2-7bca-4a0a-a6b8-f3e7db2b0745",
108+
"group_by" : "NONE",
109+
"indices" : [
110+
"my-index-0"
111+
],
112+
"source" : {
113+
"size" : 20,
114+
"query" : {
115+
"bool" : {
116+
"must" : [
117+
{
118+
"match_phrase" : {
119+
"message" : {
120+
"query" : "document",
121+
"slop" : 0,
122+
"zero_terms_query" : "NONE",
123+
"boost" : 1.0
124+
}
125+
}
126+
},
127+
{
128+
"match" : {
129+
"user.id" : {
130+
"query" : "userId",
131+
"operator" : "OR",
132+
"prefix_length" : 0,
133+
"max_expansions" : 50,
134+
"fuzzy_transpositions" : true,
135+
"lenient" : false,
136+
"zero_terms_query" : "NONE",
137+
"auto_generate_synonyms_phrase_query" : true,
138+
"boost" : 1.0
139+
}
140+
}
141+
}
142+
],
143+
"adjust_pure_negative" : true,
144+
"boost" : 1.0
145+
}
146+
}
147+
},
148+
"task_resource_usages" : [
149+
{
150+
"action" : "indices:data/read/search[phase/query]",
151+
"taskId" : 28,
152+
"parentTaskId" : 27,
153+
"nodeId" : "BBgWzu8QR0qDkR0G45aw8w",
154+
"taskResourceUsage" : {
155+
"cpu_time_in_nanos" : 22664000,
156+
"memory_in_bytes" : 6604536
157+
}
158+
},
159+
{
160+
"action" : "indices:data/read/search",
161+
"taskId" : 27,
162+
"parentTaskId" : -1,
163+
"nodeId" : "BBgWzu8QR0qDkR0G45aw8w",
164+
"taskResourceUsage" : {
165+
"cpu_time_in_nanos" : 119000,
166+
"memory_in_bytes" : 3920
167+
}
168+
}
169+
],
170+
"node_id" : "BBgWzu8QR0qDkR0G45aw8w",
171+
"phase_latency_map" : {
172+
"expand" : 0,
173+
"query" : 23,
174+
"fetch" : 0
175+
},
176+
"labels" : {
177+
"X-Opaque-Id" : "query-label-1"
178+
},
179+
"search_type" : "query_then_fetch",
180+
"total_shards" : 1,
181+
"measurements" : {
182+
"memory" : {
183+
"number" : 6608456,
184+
"count" : 1,
185+
"aggregationType" : "NONE"
186+
},
187+
"latency" : {
188+
"number" : 24,
189+
"count" : 1,
190+
"aggregationType" : "NONE"
191+
},
192+
"cpu" : {
193+
"number" : 22783000,
194+
"count" : 1,
195+
"aggregationType" : "NONE"
196+
}
197+
}
198+
},
199+
{
200+
"timestamp" : 1745021826937,
201+
"id" : "86e161d0-e982-48c2-b8da-e3a3763f2e36",
202+
"group_by" : "NONE",
203+
"indices" : [
204+
"my-index-*"
205+
],
206+
"source" : {
207+
"size" : 20,
208+
"query" : {
209+
"term" : {
210+
"user.id" : {
211+
"value" : "userId",
212+
"boost" : 1.0
213+
}
214+
}
215+
}
216+
},
217+
"task_resource_usages" : [
218+
{
219+
"action" : "indices:data/read/search[phase/query]",
220+
"taskId" : 26,
221+
"parentTaskId" : 25,
222+
"nodeId" : "BBgWzu8QR0qDkR0G45aw8w",
223+
"taskResourceUsage" : {
224+
"cpu_time_in_nanos" : 11020000,
225+
"memory_in_bytes" : 4292272
226+
}
227+
},
228+
{
229+
"action" : "indices:data/read/search",
230+
"taskId" : 25,
231+
"parentTaskId" : -1,
232+
"nodeId" : "BBgWzu8QR0qDkR0G45aw8w",
233+
"taskResourceUsage" : {
234+
"cpu_time_in_nanos" : 1032000,
235+
"memory_in_bytes" : 115816
236+
}
237+
}
238+
],
239+
"node_id" : "BBgWzu8QR0qDkR0G45aw8w",
240+
"phase_latency_map" : {
241+
"expand" : 0,
242+
"query" : 15,
243+
"fetch" : 1
244+
},
245+
"labels" : { },
246+
"search_type" : "query_then_fetch",
247+
"total_shards" : 1,
248+
"measurements" : {
249+
"memory" : {
250+
"number" : 4408088,
251+
"count" : 1,
252+
"aggregationType" : "NONE"
253+
},
254+
"latency" : {
255+
"number" : 23,
256+
"count" : 1,
257+
"aggregationType" : "NONE"
258+
},
259+
"cpu" : {
260+
"number" : 12052000,
261+
"count" : 1,
262+
"aggregationType" : "NONE"
263+
}
264+
}
265+
}
266+
]
267+
}
268+
```
269+
270+
</details>
271+
94272
If your query returns no results, ensure that top N query monitoring is enabled for the target metric type and that search requests were made within the current [time window](#configuring-the-window-size).
95273
{: .important}
96274

0 commit comments

Comments
 (0)