Skip to content
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions troubleshoot/elasticsearch.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,12 @@ If you're using {{ech}}, you can use AutoOps to monitor your cluster. AutoOps si
* [](/troubleshoot/elasticsearch/increase-cluster-shard-limit.md)
* [](/troubleshoot/elasticsearch/corruption-troubleshooting.md)

## Errors [troubleshooting-errors]

* [](/troubleshoot/elasticsearch/all-shards-failed.md)
* [](/troubleshoot/elasticsearch/failed-to-parse-field-of-type.md)
* [](/troubleshoot/elasticsearch/unable-to-retrieve-node-fs-stats.md)
* [](/troubleshoot/elasticsearch/unable-to-parse-response-body.md)

## Management [troubleshooting-management]

Expand Down
199 changes: 199 additions & 0 deletions troubleshoot/elasticsearch/all-shards-failed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
---
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: all shards failed"
---

# Fix error: All shards failed [all-shards-failed]

```console
Error: all shards failed
```

The `all shards failed` error indicates that {{es}} couldn't get a successful response from any of the shards involved in the query. Possible causes include shard allocation issues, misconfiguration, insufficient resources, or unsupported operations such as aggregating on text fields.

## Unsupported operations on text fields

The `all shards failed` error can occur when you try to sort or aggregate on `text` fields. These fields are designed for full-text search and don't support exact-value operations like sorting and aggregation.

To fix this issue, use a `.keyword` subfield:

```console
GET my-index/_search
{
"aggs": {
"names": {
"terms": {
"field": "name.keyword"
}
}
}
}
```

If no `.keyword` subfield exists, define a [multi-field](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md#types-multi-fields) mapping:

```console
PUT my-index
{
"mappings": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
```

### Metric aggregations on text fields

The `all shards failed` error can also occur when you use a metric aggregation on a text field. [Metric aggregations](elasticsearch://reference/aggregations/metrics.md) require numeric fields.

You can use a script to convert the text value to a number at query time:

```console
GET my-index/_search
{
"aggs": {
"total_cost": {
"sum": {
"script": {
"source": "Integer.parseInt(doc.cost.value)"
}
}
}
}
}
```

Or change the field mapping to a numeric type:

```console
PUT my-index
{
"mappings": {
"properties": {
"cost": {
"type": "integer"
}
}
}
}
```

## Failed shard recovery

A shard failure during recovery can prevent successful queries.

To identify the cause, check the cluster health:

```console
GET _cluster/health
```

As a last resort, you can delete the problematic index.

## Misused global aggregation

[Global aggregations](elasticsearch://reference/aggregations/search-aggregations-bucket-global-aggregation.md) must be defined at the top level of the aggregations object. Nesting can cause errors.

To fix this issue, structure the query so that the `global` aggregation appears at the top level:

```console
GET my-index/_search
{
"size": 0,
"aggs": {
"all_products": {
"global": {},
"aggs": {
"genres": {
"terms": {
"field": "cost"
}
}
}
}
}
}
```

## Reverse_nested usage errors

Using a [`reverse_nested`](elasticsearch://reference/aggregations/search-aggregations-bucket-reverse-nested-aggregation.md) aggregation outside of a `nested` context causes errors.

To fix this issue, structure the query so that the `reverse_nested` aggregation is inside a `nested` aggregation:

```console
GET my-index/_search
{
"aggs": {
"comments": {
"nested": {
"path": "comments"
},
"aggs": {
"top_usernames": {
"terms": {
"field": "comments.username"
},
"aggs": {
"comment_issue": {
"reverse_nested": {},
"aggs": {
"top_tags": {
"terms": {
"field": "tags"
}
}
}
}
}
}
}
}
}
}
```

## Further troubleshooting

Use the `_cat/shards` API to view shard status and troubleshoot further.

```console
GET _cat/shards
```

For a specific index:

```console
GET _cat/shards/my-index
```

Example output:

```console-result
my-index 5 p STARTED 0 283b 127.0.0.1 ziap
my-index 5 r UNASSIGNED
my-index 2 p STARTED 1 3.7kb 127.0.0.1 ziap
my-index 2 r UNASSIGNED
my-index 3 p STARTED 3 7.2kb 127.0.0.1 ziap
my-index 3 r UNASSIGNED
my-index 1 p STARTED 1 3.7kb 127.0.0.1 ziap
my-index 1 r UNASSIGNED
my-index 4 p STARTED 2 3.8kb 127.0.0.1 ziap
my-index 4 r UNASSIGNED
my-index 0 p STARTED 0 283b 127.0.0.1 ziap
my-index 0 r UNASSIGNED
```
12 changes: 12 additions & 0 deletions troubleshoot/elasticsearch/errors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
navigation_title: Error reference
---

# Troubleshoot common errors in {{es}}

Use the topics in this section to troubleshoot common errors in {{es}} deployments.

* [](/troubleshoot/elasticsearch/all-shards-failed.md)
* [](/troubleshoot/elasticsearch/failed-to-parse-field-of-type.md)
* [](/troubleshoot/elasticsearch/unable-to-retrieve-node-fs-stats.md)
* [](/troubleshoot/elasticsearch/unable-to-parse-response-body.md)
58 changes: 58 additions & 0 deletions troubleshoot/elasticsearch/failed-to-parse-field-of-type.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
---
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: Failed to parse field of type in document with id"
---

# Fix error: Failed to parse field [failed-to-parse-field-of-type]

```console
Error: failed to parse field [field] of type [type] in document with id [id]
```

This error occurs when you try to index a document, but one of the field values doesn't match the expected data type. {{es}} rejects the document when it encounters incompatible values, like a string in a numeric field or an invalid IP address.

To fix this issue, make sure each field value matches the data type defined in the mapping.

## Field types and mapping

When no explicit mapping exists, {{es}} uses [dynamic mappings](../../manage-data/data-store/mapping/dynamic-field-mapping.md) to infer a field's type based on the **first value indexed**.

For example, if you index:

```console
PUT test/_doc/1
{
"ip_address": "179.152.62.82",
"boolean_field": "off"
}
```

Without explicit mapping, {{es}} will treat `ip_address` and `boolean_field` as `text`, which might not be the intended result.

To avoid this, define the mapping explicitly:

```console
PUT test
{
"mappings": {
"properties": {
"ip_address": { "type": "ip" },
"boolean_field": { "type": "boolean" }
}
}
}
```

To check the data type of the field causing the error, first get the mapping:

```console
GET your-index-name/_mapping
```

Make sure the incoming data matches the expected type. If not, you'll need to fix the data or update the mapping. If necessary, create a new index with the correct mapping and reindex your data.
60 changes: 60 additions & 0 deletions troubleshoot/elasticsearch/security/token-invalid-expired.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: token invalid or expired"
---

# Fix token invalid or expired error in Elasticsearch [token-invalid-expired]

```console
Error: token expired

Error: invalid token
```

This error occurs when {{es}} receives a request containing an invalid or expired token during authentication. It's typically caused by missing, incorrect, or outdated tokens. If an invalid or expired token is used, {{es}} rejects the request.

## Invalid token

An invalid token will cause {{es}} to reject a request. Common causes include:

- The token is expired or revoked
- The token format is incorrect or malformed
- The Authorization header is missing or doesn’t start with Bearer
- The client or middleware failed to attach the token properly
- Security settings in {{es}} are misconfigured

To resolve this error:

- Verify the token and ensure it's correctly formatted and current.
- Check expiration and generate a new token if needed.
- Inspect your client and confirm the token is sent in the `Authorization` header.
- Review {{es}} settings and check that token auth is enabled:

```yaml
xpack.security.authc.token.enabled: true
```
- Use logs for details: {{es}} logs may provide context about the failure.
## Token expired
This error occurs when {{es}} receives a request containing an expired token during authentication.
To resolve this issue:
- Refresh the token, and obtain a new token using your token refresh workflow.
- Implement automatic token refresh and ensure your application is configured to refresh tokens before expiration.
- Avoid using expired tokens and do not reuse tokens after logout or expiration.
- Adjust token lifespan if needed and configure a longer token expiration in `elasticsearch.yml`, though this should be balanced against security needs:

```yaml
xpack.security.authc.token.timeout: 20m
```

Loading
Loading