Skip to content

Commit 4fc1b4a

Browse files
mromaioskderussokilfoyle
authored
docs: add 429 semantic_text troubleshooting (#2507)
This PR adds some troubleshooting information for addressing 429 responses due to `semantic_text` ingestion. --------- Co-authored-by: Kathleen DeRusso <[email protected]> Co-authored-by: David Kilfoyle <[email protected]>
1 parent ad0d071 commit 4fc1b4a

File tree

1 file changed

+24
-2
lines changed

1 file changed

+24
-2
lines changed

troubleshoot/elasticsearch/rejected-requests.md

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,13 +67,35 @@ These stats are cumulative from node startup.
6767

6868
Indexing pressure rejections appear as an `EsRejectedExecutionException`, and indicate that they were rejected due to `combined_coordinating_and_primary`, `coordinating`, `primary`, or `replica`.
6969

70-
These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](elasticsearch://reference/elasticsearch/index-settings/index-modules.md).
70+
These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](elasticsearch://reference/elasticsearch/index-settings/index-modules.md).
71+
72+
::::{note}
73+
{applies_to}`stack: ga 9.1`{applies_to}`serverless: ga`
74+
Another cause of indexing pressure rejections might be the use of the [`semantic_text`](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text) field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out of Memory (OOM) error.
75+
::::
7176

7277
See [this video](https://www.youtube.com/watch?v=QuV8QqSfc0c) for a walkthrough of diagnosing indexing pressure rejections.
7378

7479

7580
## Prevent rejected requests [prevent-rejected-requests]
7681

77-
**Fix high CPU and memory usage**
82+
### Fix high CPU and memory usage [fix-high-cpu-memory-usage]
7883

7984
If {{es}} regularly rejects requests and other tasks, your cluster likely has high CPU usage or high JVM memory pressure. For tips, see [High CPU usage](high-cpu-usage.md) and [High JVM memory pressure](high-jvm-memory-pressure.md).
85+
86+
### Fix for `semantic_text` ingestion issues [fix-semantic-text-ingestion-issues]
87+
```{applies_to}
88+
stack: ga 9.1
89+
serverless: ga
90+
```
91+
When bulk indexing documents with the `semantic_text` field type, you may encounter rejections due to high memory usage during inference processing.
92+
These rejections will appear as an `InferenceException` in your cluster logs.
93+
94+
**To resolve this issue:**
95+
96+
1. Reduce the batch size of documents in your indexing requests.
97+
2. If reducing batch size doesn't resolve the issue, then consider scaling up your machine resources.
98+
3. {applies_to}`serverless: unavailable` A last resort option is to adjust the `indexing_pressure.memory.coordinating.limit` cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur.
99+
::::{warning}
100+
This adjustment should only be considered after exhausting other options, as setting this value too high may risk Out of Memory (OOM) errors in your cluster. A cluster restart is required for this change to take effect.
101+
::::

0 commit comments

Comments
 (0)