Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions troubleshoot/elasticsearch.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,14 @@ If you're using {{ech}}, you can use AutoOps to monitor your cluster. AutoOps si
* [](/troubleshoot/elasticsearch/increase-cluster-shard-limit.md)
* [](/troubleshoot/elasticsearch/corruption-troubleshooting.md)

## Errors [troubleshooting-errors]

* [](/troubleshoot/elasticsearch/all-shards-failed.md)
* [](/troubleshoot/elasticsearch/failed-to-parse-field-of-type.md)
* [](/troubleshoot/elasticsearch/unable-to-retrieve-node-fs-stats.md)
* [](/troubleshoot/elasticsearch/unable-to-parse-response-body.md)
* [](/troubleshoot/elasticsearch/updating-number-of-replicas-to-for-indices.md)
* [](/troubleshoot/elasticsearch/memory-locking-error.md)

## Management [troubleshooting-management]

Expand Down
200 changes: 200 additions & 0 deletions troubleshoot/elasticsearch/all-shards-failed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
---
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: all shards failed"
# is mapped_pages needed for newly created docs?
---

# Fix all shards failed error [all-shards-failed]

```
Error: all shards failed
```

This error indicates that {{es}} failed to retrieve a response from any shard involved in a query. This can result from shard allocation issues, misconfiguration, insufficient resources, or unsupported operations such as aggregating on text fields.

## Improper use of text fields

Text fields aren't optimized for operations like sorting or aggregations by default. Attempting these operations may trigger the error.

To fix, use the `.keyword` sub-field:

```console
GET my-index/_search
{
"aggs": {
"names": {
"terms": {
"field": "name.keyword"
}
}
}
}
```

If no `.keyword` sub-field exists, update the mapping to handle [multi-fields](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md):

```console
PUT my-index
{
"mappings": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
```

## Metric aggregations on text fields

[Metric aggregations](elasticsearch://reference/aggregations/metrics.md) require numeric fields. Attempting them on text fields will fail.

Use a script to convert the text to numeric:

```console
GET my-index/_search
{
"aggs": {
"total_cost": {
"sum": {
"script": {
"source": "Integer.parseInt(doc.cost.value)"
}
}
}
}
}
```

Or change the field mapping to a numeric type:

```console
PUT my-index
{
"mappings": {
"properties": {
"cost": {
"type": "integer"
}
}
}
}
```

## Failed shard recovery

A shard failure during recovery can prevent successful queries.

To confirm, check cluster health:

```console
GET _cluster/health
```

Identify and resolve the cause. If necessary, and as a last resort, delete the problematic index.

## Misused global aggregation

[Global aggregations](elasticsearch://reference/aggregations/search-aggregations-bucket-global-aggregation.md) must be top-level. Nesting them incorrectly causes errors.

To fix, structure the query so the `global` aggregation is top-level:

```console
GET my-index/_search
{
"size": 0,
"aggs": {
"all_products": {
"global": {},
"aggs": {
"genres": {
"terms": {
"field": "cost"
}
}
}
}
}
}
```

## Reverse_nested usage errors

The [reverse_nested](elasticsearch://reference/aggregations/search-aggregations-bucket-reverse-nested-aggregation.md) aggregation must appear within a `nested` context.

To fix, structure the query so the `reverse_nested` aggregation is within a `nested` context:

```console
GET my-index/_search
{
"aggs": {
"comments": {
"nested": {
"path": "comments"
},
"aggs": {
"top_usernames": {
"terms": {
"field": "comments.username"
},
"aggs": {
"comment_issue": {
"reverse_nested": {},
"aggs": {
"top_tags": {
"terms": {
"field": "tags"
}
}
}
}
}
}
}
}
}
}
```

## Further troubleshooting

Use the `_cat/shards` API to view shard status and troubleshoot further.

```console
GET _cat/shards
```

For a specific index:

```console
GET _cat/shards/my-index
```

Example output:

```console-result
my-index 5 p STARTED 0 283b 127.0.0.1 ziap
my-index 5 r UNASSIGNED
my-index 2 p STARTED 1 3.7kb 127.0.0.1 ziap
my-index 2 r UNASSIGNED
my-index 3 p STARTED 3 7.2kb 127.0.0.1 ziap
my-index 3 r UNASSIGNED
my-index 1 p STARTED 1 3.7kb 127.0.0.1 ziap
my-index 1 r UNASSIGNED
my-index 4 p STARTED 2 3.8kb 127.0.0.1 ziap
my-index 4 r UNASSIGNED
my-index 0 p STARTED 0 283b 127.0.0.1 ziap
my-index 0 r UNASSIGNED
```
1 change: 1 addition & 0 deletions troubleshoot/elasticsearch/data.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Use the topics in this section to troubleshoot data-related issues in your {{es}
* [](/troubleshoot/elasticsearch/increase-cluster-shard-limit.md)
* [](/troubleshoot/elasticsearch/corruption-troubleshooting.md)


## Additional resources
* [](/troubleshoot/elasticsearch/fix-watermark-errors.md)
* [Troubleshooting overview](/troubleshoot/index.md)
Expand Down
14 changes: 14 additions & 0 deletions troubleshoot/elasticsearch/errors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
navigation_title: Errors
---

# Troubleshoot errors in {{es}}

Use the topics in this section to troubleshoot errors in your {{es}} deployments.

* [](/troubleshoot/elasticsearch/all-shards-failed.md)
* [](/troubleshoot/elasticsearch/failed-to-parse-field-of-type.md)
* [](/troubleshoot/elasticsearch/unable-to-retrieve-node-fs-stats.md)
* [](/troubleshoot/elasticsearch/unable-to-parse-response-body.md)
* [](/troubleshoot/elasticsearch/updating-number-of-replicas-to-for-indices.md)
* [](/troubleshoot/elasticsearch/memory-locking-error.md)
65 changes: 65 additions & 0 deletions troubleshoot/elasticsearch/failed-to-parse-field-of-type.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: failed to parse field of type in document with id"
# is mapped_pages needed for newly created docs?
---

# Fix failed to parse field of type in document with id [failed-to-parse-field-of-type]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Fix failed to parse field of type in document with id [failed-to-parse-field-of-type]
# Fix error: Failed to parse field [failed-to-parse-field-of-type]


```console
Error: failed to parse field [field] of type [type] in document with id [id]
```

This error occurs when you try to index a document into an existing index, and one of your fields' values doesn't match the expected data type. {{es}} rejects the document when it encounters incompatible values, like a string in a numeric field or an invalid IP address.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This error occurs when you try to index a document into an existing index, and one of your fields' values doesn't match the expected data type. {{es}} rejects the document when it encounters incompatible values, like a string in a numeric field or an invalid IP address.
This error occurs when you try to index a document, but one of the field values doesn't match the expected data type. {{es}} rejects the document when it encounters incompatible values, like a string in a numeric field or an invalid IP address.


To fix, ensure the field's value matches the expected data type in the mapping.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To fix, ensure the field's value matches the expected data type in the mapping.
To fix this issue, make sure each field value matches the data type defined in the mapping.


## Understand field types and mapping

By default, {{es}} uses [dynamic mappings](../../manage-data/data-store/mapping/dynamic-field-mapping.md) to infer a field's type based on the **first value indexed** when no explicit mapping exists.

For example, if you index:

```console
PUT test/_doc/1
{
"ip_address": "179.152.62.82",
"boolean_field": "off"
}
```

Without explicit mapping, {{es}} will treat `ip_address` and `boolean_field` as `text` which may not be what is intended.

To avoid this, define your mapping explicitly:

```console
PUT test
{
"mappings": {
"properties": {
"ip_address": { "type": "ip" },
"boolean_field": { "type": "boolean" }
}
}
}
```

## Troubleshoot and resolve the error

1. Check the mapping of your index:

```console
GET your-index-name/_mapping
```

2. Confirm the data type of the field causing the error.

3. Ensure the incoming data matches the expected type.

4. If necessary, create a new index with the correct mapping and reindex your data.
60 changes: 60 additions & 0 deletions troubleshoot/elasticsearch/memory-locking-error.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This topic doesn't actually tell the user how to resolve the error. Instead, it points to a configuration doc that includes some troubleshooting info at the end. Consider pulling the troubleshooting info out of the existing doc and adding it to this new topic (and removing the how-to info from this topic). I'd probably do this in a separate PR and just leave this topic out for now.

Without some kind of revamping, this topic isn't really viable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This topic was one I was sitting on the fence on, but I do agree, so I'll remove it.

applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: memory locking requested for elasticsearch process but memory is not locked"
# is mapped_pages needed for newly created docs?
---

# Fix memory locking error in Elasticsearch [memory-locking-error]

```console
Error: memory locking requested for elasticsearch process but memory is not locked
```


This error indicates that {{es}} attempted to lock its memory to prevent swapping but failed. Swapping can severely impact performance and stability by introducing large garbage collection (GC) pauses.

You can fix this by reviewing your memory swapping options, and adjust as needed.

## What it means

{{es}} uses the `bootstrap.memory_lock: true` setting to request that its memory be locked into RAM, preventing the OS from swapping it to disk. If this lock fails due to missing system permissions or improper configuration, {{es}} logs this error.

## How to resolve it

1. **Enable memory lock in configuration**:

Edit `elasticsearch.yml` and set:

```yaml
bootstrap.memory_lock: true
```
2. **Verify the memory lock status**:
Run the following:
```console
GET _nodes?filter_path=**.mlockall
```

A successful configuration will return:

```json
{
"nodes" : {
"<node_id>" : {
"process" : {
"mlockall" : true
}
}
}
}
```

If the value is `false`, further [system-level configuration](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html) is required.
Loading
Loading