From de84f8d23e03a1d28e3e03dc3b1f8fe6a09ad05b Mon Sep 17 00:00:00 2001 From: Michail Romaios Date: Tue, 12 Aug 2025 17:41:03 +0200 Subject: [PATCH 1/4] docs: add 429 semantic_text troubleshooting note --- .../common-issues/rejected-requests.asciidoc | 23 +++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc b/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc index a0dcef45adc4b..c9e328468de9e 100644 --- a/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc +++ b/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc @@ -63,7 +63,7 @@ rejections, use the <>. GET _nodes/stats?human&filter_path=nodes.*.indexing_pressure ---- -These stats are cumulative from node startup. +These stats are cumulative from node startup. Indexing pressure rejections appear as an `EsRejectedExecutionException`, and indicate that they were rejected due @@ -73,12 +73,31 @@ These errors are often related to <>, <> sizing, or the ingest target's <>. +NOTE: Another cause of indexing pressure rejections might be the use of the <> field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out of Memory (OOM) error. + [discrete] [[prevent-rejected-requests]] ==== Prevent rejected requests -**Fix high CPU and memory usage** +[discrete] +[[fix-high-cpu-and-memory-usage]] +===== Fix high CPU and memory usage If {es} regularly rejects requests and other tasks, your cluster likely has high CPU usage or high JVM memory pressure. For tips, see <> and <>. + +[discrete] +[[fix-semantic-text-ingestion-issues]] +===== Fix for `semantic_text` ingestion issues + +When bulk indexing documents with the `semantic_text` field type, you may encounter rejections due to high memory usage during inference processing. +These rejections will appear as an `InferenceException` in your cluster logs. + +**To resolve this issue:** + +1. Reduce the batch size of documents in your indexing requests. +2. If reducing batch size doesn't resolve the issue, then consider scaling up your machine resources. +3. A last resort option is to adjust the `indexing_pressure.memory.coordinating.limit` cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur. + +WARNING: This adjustment should only be considered after exhausting other options, as setting this value too high may risk Out of Memory (OOM) errors in your cluster. A cluster restart is required for this change to take effect. From 1257b6128b0d3f066ed9d7d7c0f5b31b1941cc55 Mon Sep 17 00:00:00 2001 From: Michail Romaios Date: Wed, 13 Aug 2025 10:18:08 +0200 Subject: [PATCH 2/4] fix semantic_text reference --- .../troubleshooting/common-issues/rejected-requests.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc b/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc index c9e328468de9e..8b3d28af278dc 100644 --- a/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc +++ b/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc @@ -73,7 +73,7 @@ These errors are often related to <>, <> sizing, or the ingest target's <>. -NOTE: Another cause of indexing pressure rejections might be the use of the <> field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out of Memory (OOM) error. +NOTE: Another cause of indexing pressure rejections might be the use of the <> field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out of Memory (OOM) error. [discrete] [[prevent-rejected-requests]] From 1cba5a47393e55d39825fcfe04b7949aa8096f52 Mon Sep 17 00:00:00 2001 From: Michail Romaios Date: Wed, 13 Aug 2025 11:03:42 +0200 Subject: [PATCH 3/4] format semantic_text ref --- .../troubleshooting/common-issues/rejected-requests.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc b/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc index 8b3d28af278dc..9c26a4f6b41bc 100644 --- a/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc +++ b/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc @@ -73,7 +73,7 @@ These errors are often related to <>, <> sizing, or the ingest target's <>. -NOTE: Another cause of indexing pressure rejections might be the use of the <> field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out of Memory (OOM) error. +NOTE: Another cause of indexing pressure rejections might be the use of the <> field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out of Memory (OOM) error. [discrete] [[prevent-rejected-requests]] From 9fdbc36ceb781b0d8ad5a752a7f2cec4a9cb1d28 Mon Sep 17 00:00:00 2001 From: Michail Romaios Date: Wed, 13 Aug 2025 11:57:26 +0200 Subject: [PATCH 4/4] fix semantic_text reference --- .../troubleshooting/common-issues/rejected-requests.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc b/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc index 9c26a4f6b41bc..14c2be15d1413 100644 --- a/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc +++ b/docs/reference/troubleshooting/common-issues/rejected-requests.asciidoc @@ -73,7 +73,7 @@ These errors are often related to <>, <> sizing, or the ingest target's <>. -NOTE: Another cause of indexing pressure rejections might be the use of the <> field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out of Memory (OOM) error. +NOTE: Another cause of indexing pressure rejections might be the use of the <> field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out of Memory (OOM) error. [discrete] [[prevent-rejected-requests]]