You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This PR adds some troubleshooting information for addressing 429
responses due to `semantic_text` ingestion.
---------
Co-authored-by: Kathleen DeRusso <[email protected]>
Co-authored-by: David Kilfoyle <[email protected]>
Copy file name to clipboardExpand all lines: troubleshoot/elasticsearch/rejected-requests.md
+24-2Lines changed: 24 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -67,13 +67,35 @@ These stats are cumulative from node startup.
67
67
68
68
Indexing pressure rejections appear as an `EsRejectedExecutionException`, and indicate that they were rejected due to `combined_coordinating_and_primary`, `coordinating`, `primary`, or `replica`.
69
69
70
-
These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](elasticsearch://reference/elasticsearch/index-settings/index-modules.md).
70
+
These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](elasticsearch://reference/elasticsearch/index-settings/index-modules.md).
71
+
72
+
::::{note}
73
+
{applies_to}`stack: ga 9.1`{applies_to}`serverless: ga`
74
+
Another cause of indexing pressure rejections might be the use of the [`semantic_text`](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text) field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out of Memory (OOM) error.
75
+
::::
71
76
72
77
See [this video](https://www.youtube.com/watch?v=QuV8QqSfc0c) for a walkthrough of diagnosing indexing pressure rejections.
### Fix high CPU and memory usage[fix-high-cpu-memory-usage]
78
83
79
84
If {{es}} regularly rejects requests and other tasks, your cluster likely has high CPU usage or high JVM memory pressure. For tips, see [High CPU usage](high-cpu-usage.md) and [High JVM memory pressure](high-jvm-memory-pressure.md).
85
+
86
+
### Fix for `semantic_text` ingestion issues [fix-semantic-text-ingestion-issues]
87
+
```{applies_to}
88
+
stack: ga 9.1
89
+
serverless: ga
90
+
```
91
+
When bulk indexing documents with the `semantic_text` field type, you may encounter rejections due to high memory usage during inference processing.
92
+
These rejections will appear as an `InferenceException` in your cluster logs.
93
+
94
+
**To resolve this issue:**
95
+
96
+
1. Reduce the batch size of documents in your indexing requests.
97
+
2. If reducing batch size doesn't resolve the issue, then consider scaling up your machine resources.
98
+
3. {applies_to}`serverless: unavailable` A last resort option is to adjust the `indexing_pressure.memory.coordinating.limit` cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur.
99
+
::::{warning}
100
+
This adjustment should only be considered after exhausting other options, as setting this value too high may risk Out of Memory (OOM) errors in your cluster. A cluster restart is required for this change to take effect.
0 commit comments