Skip to content
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
200 changes: 200 additions & 0 deletions troubleshoot/elasticsearch/all-shards-failed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
---
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: All shards failed"
# is mapped_pages needed for newly created docs?
---

# Fix error: All shards failed [all-shards-failed]

```console
Error: all shards failed
```

The `all shards failed` error indicates that {{es}} couldn't get a successful response from any of the shards involved in the query. Possible causes include shard allocation issues, misconfiguration, insufficient resources, or unsupported operations such as aggregating on text fields.

## Unsupported operations on text fields

The `all shards failed` error can occur when you try to sort or aggregate on `text` fields. These fields are designed for full-text search and don't support exact-value operations like sorting and aggregation.

To fix this issue, use a `.keyword` subfield:

```console
GET my-index/_search
{
"aggs": {
"names": {
"terms": {
"field": "name.keyword"
}
}
}
}
```

If no `.keyword` subfield exists, define a [multi-field](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md#types-multi-fields) mapping:

```console
PUT my-index
{
"mappings": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
```

### Metric aggregations on text fields

The `all shards failed` error can also occur when you use a metric aggregation on a text field. [Metric aggregations](elasticsearch://reference/aggregations/metrics.md) require numeric fields.

You can use a script to convert the text value to a number at query time:

```console
GET my-index/_search
{
"aggs": {
"total_cost": {
"sum": {
"script": {
"source": "Integer.parseInt(doc.cost.value)"
}
}
}
}
}
```

Or change the field mapping to a numeric type:

```console
PUT my-index
{
"mappings": {
"properties": {
"cost": {
"type": "integer"
}
}
}
}
```

## Failed shard recovery

A shard failure during recovery can prevent successful queries.

To identify the cause, check the cluster health:

```console
GET _cluster/health
```

As a last resort, you can delete the problematic index.

## Misused global aggregation

[Global aggregations](elasticsearch://reference/aggregations/search-aggregations-bucket-global-aggregation.md) must be defined at the top level of the aggregations object. Nesting can cause errors.

To fix this issue, structure the query so that the `global` aggregation appears at the top level:

```console
GET my-index/_search
{
"size": 0,
"aggs": {
"all_products": {
"global": {},
"aggs": {
"genres": {
"terms": {
"field": "cost"
}
}
}
}
}
}
```

## Reverse_nested usage errors

Using a [`reverse_nested`](elasticsearch://reference/aggregations/search-aggregations-bucket-reverse-nested-aggregation.md) aggregation outside of a `nested` context causes errors.

To fix this issue, structure the query so that the `reverse_nested` aggregation is inside a `nested` aggregation:

```console
GET my-index/_search
{
"aggs": {
"comments": {
"nested": {
"path": "comments"
},
"aggs": {
"top_usernames": {
"terms": {
"field": "comments.username"
},
"aggs": {
"comment_issue": {
"reverse_nested": {},
"aggs": {
"top_tags": {
"terms": {
"field": "tags"
}
}
}
}
}
}
}
}
}
}
```

## Further troubleshooting

Use the `_cat/shards` API to view shard status and troubleshoot further.

```console
GET _cat/shards
```

For a specific index:

```console
GET _cat/shards/my-index
```

Example output:

```console-result
my-index 5 p STARTED 0 283b 127.0.0.1 ziap
my-index 5 r UNASSIGNED
my-index 2 p STARTED 1 3.7kb 127.0.0.1 ziap
my-index 2 r UNASSIGNED
my-index 3 p STARTED 3 7.2kb 127.0.0.1 ziap
my-index 3 r UNASSIGNED
my-index 1 p STARTED 1 3.7kb 127.0.0.1 ziap
my-index 1 r UNASSIGNED
my-index 4 p STARTED 2 3.8kb 127.0.0.1 ziap
my-index 4 r UNASSIGNED
my-index 0 p STARTED 0 283b 127.0.0.1 ziap
my-index 0 r UNASSIGNED
```
1 change: 1 addition & 0 deletions troubleshoot/elasticsearch/data.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Use the topics in this section to troubleshoot data-related issues in your {{es}
* [](/troubleshoot/elasticsearch/increase-cluster-shard-limit.md)
* [](/troubleshoot/elasticsearch/corruption-troubleshooting.md)


## Additional resources
* [](/troubleshoot/elasticsearch/fix-watermark-errors.md)
* [Troubleshooting overview](/troubleshoot/index.md)
Expand Down
14 changes: 14 additions & 0 deletions troubleshoot/elasticsearch/errors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
navigation_title: Error reference
---

# Troubleshoot common errors in {{es}}

Use the topics in this section to troubleshoot common errors in {{es}} deployments.

* [](/troubleshoot/elasticsearch/all-shards-failed.md)
* [](/troubleshoot/elasticsearch/failed-to-parse-field-of-type.md)
* [](/troubleshoot/elasticsearch/unable-to-retrieve-node-fs-stats.md)
* [](/troubleshoot/elasticsearch/unable-to-parse-response-body.md)
* [](/troubleshoot/elasticsearch/updating-number-of-replicas-to-for-indices.md)
* [](/troubleshoot/elasticsearch/memory-locking-error.md)
65 changes: 65 additions & 0 deletions troubleshoot/elasticsearch/failed-to-parse-field-of-type.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: failed to parse field of type in document with id"
# is mapped_pages needed for newly created docs?
---

# Fix error: Failed to parse field [failed-to-parse-field-of-type]

```console
Error: failed to parse field [field] of type [type] in document with id [id]
```

This error occurs when you try to index a document, but one of the field values doesn't match the expected data type. {{es}} rejects the document when it encounters incompatible values, like a string in a numeric field or an invalid IP address.

To fix this issue, make sure each field value matches the data type defined in the mapping.

## Field types and mapping

When no explicit mapping exists, {{es}} uses [dynamic mappings](../../manage-data/data-store/mapping/dynamic-field-mapping.md) to infer a field's type based on the **first value indexed**.

For example, if you index:

```console
PUT test/_doc/1
{
"ip_address": "179.152.62.82",
"boolean_field": "off"
}
```

Without explicit mapping, {{es}} will treat `ip_address` and `boolean_field` as `text`, which might not be the intended result.

To avoid this, define the mapping explicitly:

```console
PUT test
{
"mappings": {
"properties": {
"ip_address": { "type": "ip" },
"boolean_field": { "type": "boolean" }
}
}
}
```

## Troubleshoot and resolve the error

1. Check the mapping of your index:

```console
GET your-index-name/_mapping
```

2. Confirm the data type of the field causing the error.

3. Ensure the incoming data matches the expected type.

4. If necessary, create a new index with the correct mapping and reindex your data.
59 changes: 59 additions & 0 deletions troubleshoot/elasticsearch/memory-locking-error.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This topic doesn't actually tell the user how to resolve the error. Instead, it points to a configuration doc that includes some troubleshooting info at the end. Consider pulling the troubleshooting info out of the existing doc and adding it to this new topic (and removing the how-to info from this topic). I'd probably do this in a separate PR and just leave this topic out for now.

Without some kind of revamping, this topic isn't really viable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This topic was one I was sitting on the fence on, but I do agree, so I'll remove it.

applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: memory locking requested for elasticsearch process but memory is not locked"
---

# Fix memory locking error in Elasticsearch [memory-locking-error]

```console
Error: memory locking requested for elasticsearch process but memory is not locked
```


This error indicates that {{es}} attempted to lock its memory to prevent swapping but failed. Swapping can severely impact performance and stability by introducing large garbage collection (GC) pauses.

You can fix this by reviewing your memory swapping options, and adjust as needed.

## What it means

{{es}} uses the `bootstrap.memory_lock: true` setting to request that its memory be locked into RAM, preventing the OS from swapping it to disk. If this lock fails due to missing system permissions or improper configuration, {{es}} logs this error.

## How to resolve it

1. **Enable memory lock in configuration**:

Edit `elasticsearch.yml` and set:

```yaml
bootstrap.memory_lock: true
```
2. **Verify the memory lock status**:
Run the following:
```console
GET _nodes?filter_path=**.mlockall
```

A successful configuration will return:

```json
{
"nodes" : {
"<node_id>" : {
"process" : {
"mlockall" : true
}
}
}
}
```

If the value is `false`, further [system-level configuration](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html) is required.
45 changes: 45 additions & 0 deletions troubleshoot/elasticsearch/security/invalid-token.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: invalid token"
# is mapped_pages needed for newly created docs?
---

# Fix invalid token error [invalid-token]

```console
Error: invalid token
```

This error occurs when {{es}} receives a request containing an invalid or [expired token](token-expired.md) during authentication. It's typically caused by missing, incorrect, or outdated tokens.

**Symptoms**

With security enabled in {{es}}, clients must authenticate using valid tokens. If a token is invalid or expired, the request is rejected.

This can be caused by

- Expired or revoked token
- Incorrect or malformed token format
- Missing `Authorization` header or wrong scheme (e.g., missing `Bearer` prefix)
- Token not properly attached by client or middleware
- Misconfigured security settings in {{es}}

**Resolution**

1. **Verify the token** – Ensure it’s correctly formatted and current.
2. **Check expiration** – Generate a new token if needed.
3. **Inspect your client** – Confirm the token is sent in the `Authorization` header.
4. **Review {{es}} settings** – Check that token auth is enabled:

```yaml
xpack.security.authc.token.enabled: true
```
5. **Use logs for details** – {{es}} logs may provide context about the failure.
Loading
Loading