-
Notifications
You must be signed in to change notification settings - Fork 159
[DOCS] Opster Migration: Add errors to troubleshooting pages #1116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 11 commits
b39271a
af93f7c
ef42b33
0d4ece4
2c139cb
ef1386e
821f873
cf6b31d
4e8b542
3aa17fc
b0a7c49
2bb12d6
9c3da7c
15f8b70
4c0dc4b
983d746
078c46f
ac184da
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,200 @@ | ||
--- | ||
applies_to: | ||
stack: | ||
deployment: | ||
eck: | ||
ess: | ||
ece: | ||
self: | ||
navigation_title: "Error: All shards failed" | ||
# is mapped_pages needed for newly created docs? | ||
--- | ||
|
||
# Fix error: All shards failed [all-shards-failed] | ||
|
||
```console | ||
Error: all shards failed | ||
``` | ||
|
||
The `all shards failed` error indicates that {{es}} couldn't get a successful response from any of the shards involved in the query. Possible causes include shard allocation issues, misconfiguration, insufficient resources, or unsupported operations such as aggregating on text fields. | ||
|
||
## Unsupported operations on text fields | ||
|
||
The `all shards failed` error can occur when you try to sort or aggregate on `text` fields. These fields are designed for full-text search and don't support exact-value operations like sorting and aggregation. | ||
|
||
To fix this issue, use a `.keyword` subfield: | ||
|
||
```console | ||
GET my-index/_search | ||
{ | ||
"aggs": { | ||
"names": { | ||
"terms": { | ||
"field": "name.keyword" | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
|
||
If no `.keyword` subfield exists, define a [multi-field](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md#types-multi-fields) mapping: | ||
|
||
```console | ||
PUT my-index | ||
{ | ||
"mappings": { | ||
"properties": { | ||
"name": { | ||
"type": "text", | ||
"fields": { | ||
"keyword": { | ||
"type": "keyword" | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
|
||
### Metric aggregations on text fields | ||
|
||
The `all shards failed` error can also occur when you use a metric aggregation on a text field. [Metric aggregations](elasticsearch://reference/aggregations/metrics.md) require numeric fields. | ||
|
||
You can use a script to convert the text value to a number at query time: | ||
|
||
```console | ||
GET my-index/_search | ||
{ | ||
"aggs": { | ||
"total_cost": { | ||
"sum": { | ||
"script": { | ||
"source": "Integer.parseInt(doc.cost.value)" | ||
} | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
|
||
Or change the field mapping to a numeric type: | ||
|
||
```console | ||
PUT my-index | ||
{ | ||
"mappings": { | ||
"properties": { | ||
"cost": { | ||
"type": "integer" | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
|
||
## Failed shard recovery | ||
|
||
A shard failure during recovery can prevent successful queries. | ||
|
||
To identify the cause, check the cluster health: | ||
|
||
```console | ||
GET _cluster/health | ||
``` | ||
|
||
As a last resort, you can delete the problematic index. | ||
|
||
## Misused global aggregation | ||
|
||
[Global aggregations](elasticsearch://reference/aggregations/search-aggregations-bucket-global-aggregation.md) must be defined at the top level of the aggregations object. Nesting can cause errors. | ||
|
||
To fix this issue, structure the query so that the `global` aggregation appears at the top level: | ||
|
||
```console | ||
GET my-index/_search | ||
{ | ||
"size": 0, | ||
"aggs": { | ||
"all_products": { | ||
"global": {}, | ||
"aggs": { | ||
"genres": { | ||
"terms": { | ||
"field": "cost" | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
|
||
## Reverse_nested usage errors | ||
|
||
Using a [`reverse_nested`](elasticsearch://reference/aggregations/search-aggregations-bucket-reverse-nested-aggregation.md) aggregation outside of a `nested` context causes errors. | ||
|
||
To fix this issue, structure the query so that the `reverse_nested` aggregation is inside a `nested` aggregation: | ||
|
||
```console | ||
GET my-index/_search | ||
{ | ||
"aggs": { | ||
"comments": { | ||
"nested": { | ||
"path": "comments" | ||
}, | ||
"aggs": { | ||
"top_usernames": { | ||
"terms": { | ||
"field": "comments.username" | ||
}, | ||
"aggs": { | ||
"comment_issue": { | ||
"reverse_nested": {}, | ||
"aggs": { | ||
"top_tags": { | ||
"terms": { | ||
"field": "tags" | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
|
||
## Further troubleshooting | ||
|
||
Use the `_cat/shards` API to view shard status and troubleshoot further. | ||
|
||
```console | ||
GET _cat/shards | ||
``` | ||
|
||
For a specific index: | ||
|
||
```console | ||
GET _cat/shards/my-index | ||
``` | ||
|
||
Example output: | ||
|
||
```console-result | ||
my-index 5 p STARTED 0 283b 127.0.0.1 ziap | ||
my-index 5 r UNASSIGNED | ||
my-index 2 p STARTED 1 3.7kb 127.0.0.1 ziap | ||
my-index 2 r UNASSIGNED | ||
my-index 3 p STARTED 3 7.2kb 127.0.0.1 ziap | ||
my-index 3 r UNASSIGNED | ||
my-index 1 p STARTED 1 3.7kb 127.0.0.1 ziap | ||
my-index 1 r UNASSIGNED | ||
my-index 4 p STARTED 2 3.8kb 127.0.0.1 ziap | ||
my-index 4 r UNASSIGNED | ||
my-index 0 p STARTED 0 283b 127.0.0.1 ziap | ||
my-index 0 r UNASSIGNED | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
--- | ||
navigation_title: Error reference | ||
--- | ||
|
||
# Troubleshoot common errors in {{es}} | ||
|
||
Use the topics in this section to troubleshoot common errors in {{es}} deployments. | ||
|
||
* [](/troubleshoot/elasticsearch/all-shards-failed.md) | ||
* [](/troubleshoot/elasticsearch/failed-to-parse-field-of-type.md) | ||
* [](/troubleshoot/elasticsearch/unable-to-retrieve-node-fs-stats.md) | ||
* [](/troubleshoot/elasticsearch/unable-to-parse-response-body.md) | ||
* [](/troubleshoot/elasticsearch/updating-number-of-replicas-to-for-indices.md) | ||
* [](/troubleshoot/elasticsearch/memory-locking-error.md) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
--- | ||
applies_to: | ||
stack: | ||
deployment: | ||
eck: | ||
ess: | ||
ece: | ||
self: | ||
navigation_title: "Error: failed to parse field of type in document with id" | ||
thekofimensah marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
# is mapped_pages needed for newly created docs? | ||
--- | ||
|
||
# Fix error: Failed to parse field [failed-to-parse-field-of-type] | ||
|
||
```console | ||
Error: failed to parse field [field] of type [type] in document with id [id] | ||
``` | ||
|
||
This error occurs when you try to index a document, but one of the field values doesn't match the expected data type. {{es}} rejects the document when it encounters incompatible values, like a string in a numeric field or an invalid IP address. | ||
|
||
To fix this issue, make sure each field value matches the data type defined in the mapping. | ||
|
||
## Field types and mapping | ||
|
||
When no explicit mapping exists, {{es}} uses [dynamic mappings](../../manage-data/data-store/mapping/dynamic-field-mapping.md) to infer a field's type based on the **first value indexed**. | ||
|
||
For example, if you index: | ||
|
||
```console | ||
PUT test/_doc/1 | ||
{ | ||
"ip_address": "179.152.62.82", | ||
"boolean_field": "off" | ||
} | ||
``` | ||
|
||
Without explicit mapping, {{es}} will treat `ip_address` and `boolean_field` as `text`, which might not be the intended result. | ||
|
||
To avoid this, define the mapping explicitly: | ||
|
||
```console | ||
PUT test | ||
{ | ||
"mappings": { | ||
"properties": { | ||
"ip_address": { "type": "ip" }, | ||
"boolean_field": { "type": "boolean" } | ||
} | ||
} | ||
} | ||
``` | ||
|
||
## Troubleshoot and resolve the error | ||
|
||
1. Check the mapping of your index: | ||
thekofimensah marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
```console | ||
GET your-index-name/_mapping | ||
``` | ||
thekofimensah marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
2. Confirm the data type of the field causing the error. | ||
|
||
3. Ensure the incoming data matches the expected type. | ||
|
||
4. If necessary, create a new index with the correct mapping and reindex your data. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
--- | ||
|
||
applies_to: | ||
stack: | ||
deployment: | ||
eck: | ||
ess: | ||
ece: | ||
self: | ||
navigation_title: "Error: memory locking requested for elasticsearch process but memory is not locked" | ||
thekofimensah marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
--- | ||
|
||
# Fix memory locking error in Elasticsearch [memory-locking-error] | ||
|
||
```console | ||
Error: memory locking requested for elasticsearch process but memory is not locked | ||
``` | ||
|
||
|
||
This error indicates that {{es}} attempted to lock its memory to prevent swapping but failed. Swapping can severely impact performance and stability by introducing large garbage collection (GC) pauses. | ||
|
||
You can fix this by reviewing your memory swapping options, and adjust as needed. | ||
|
||
## What it means | ||
|
||
{{es}} uses the `bootstrap.memory_lock: true` setting to request that its memory be locked into RAM, preventing the OS from swapping it to disk. If this lock fails due to missing system permissions or improper configuration, {{es}} logs this error. | ||
|
||
## How to resolve it | ||
|
||
1. **Enable memory lock in configuration**: | ||
|
||
Edit `elasticsearch.yml` and set: | ||
|
||
```yaml | ||
bootstrap.memory_lock: true | ||
``` | ||
2. **Verify the memory lock status**: | ||
Run the following: | ||
```console | ||
GET _nodes?filter_path=**.mlockall | ||
``` | ||
|
||
A successful configuration will return: | ||
|
||
```json | ||
{ | ||
"nodes" : { | ||
"<node_id>" : { | ||
"process" : { | ||
"mlockall" : true | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
|
||
If the value is `false`, further [system-level configuration](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html) is required. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
--- | ||
applies_to: | ||
stack: | ||
deployment: | ||
eck: | ||
ess: | ||
ece: | ||
self: | ||
navigation_title: "Error: invalid token" | ||
thekofimensah marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
# is mapped_pages needed for newly created docs? | ||
--- | ||
|
||
# Fix invalid token error [invalid-token] | ||
thekofimensah marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
```console | ||
Error: invalid token | ||
``` | ||
|
||
thekofimensah marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
This error occurs when {{es}} receives a request containing an invalid or [expired token](token-expired.md) during authentication. It's typically caused by missing, incorrect, or outdated tokens. | ||
|
||
**Symptoms** | ||
|
||
With security enabled in {{es}}, clients must authenticate using valid tokens. If a token is invalid or expired, the request is rejected. | ||
|
||
This can be caused by | ||
|
||
- Expired or revoked token | ||
- Incorrect or malformed token format | ||
- Missing `Authorization` header or wrong scheme (e.g., missing `Bearer` prefix) | ||
- Token not properly attached by client or middleware | ||
- Misconfigured security settings in {{es}} | ||
|
||
**Resolution** | ||
|
||
1. **Verify the token** – Ensure it’s correctly formatted and current. | ||
2. **Check expiration** – Generate a new token if needed. | ||
3. **Inspect your client** – Confirm the token is sent in the `Authorization` header. | ||
4. **Review {{es}} settings** – Check that token auth is enabled: | ||
|
||
```yaml | ||
xpack.security.authc.token.enabled: true | ||
``` | ||
5. **Use logs for details** – {{es}} logs may provide context about the failure. | ||
Uh oh!
There was an error while loading. Please reload this page.