From 767e30f6ea7932ff6801ad49e1d1a8c4c999e71c Mon Sep 17 00:00:00 2001 From: Michail Romaios Date: Mon, 11 Aug 2025 17:23:12 +0200 Subject: [PATCH 1/8] docs: add 429 semantic_text troubleshooting --- troubleshoot/elasticsearch/rejected-requests.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/troubleshoot/elasticsearch/rejected-requests.md b/troubleshoot/elasticsearch/rejected-requests.md index 21699a351a..fc8b84e825 100644 --- a/troubleshoot/elasticsearch/rejected-requests.md +++ b/troubleshoot/elasticsearch/rejected-requests.md @@ -67,7 +67,9 @@ These stats are cumulative from node startup. Indexing pressure rejections appear as an `EsRejectedExecutionException`, and indicate that they were rejected due to `combined_coordinating_and_primary`, `coordinating`, `primary`, or `replica`. -These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](elasticsearch://reference/elasticsearch/index-settings/index-modules.md). +These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](elasticsearch://reference/elasticsearch/index-settings/index-modules.md).¹ + +¹ : Starting from versions 8.19+ and 9.1+, indexing to a semantic_text field can also cause 429 errors if the batch may otherwise incur an Out Of Memory (OOM) error. See [this video](https://www.youtube.com/watch?v=QuV8QqSfc0c) for a walkthrough of diagnosing indexing pressure rejections. @@ -77,3 +79,15 @@ See [this video](https://www.youtube.com/watch?v=QuV8QqSfc0c) for a walkthrough **Fix high CPU and memory usage** If {{es}} regularly rejects requests and other tasks, your cluster likely has high CPU usage or high JVM memory pressure. For tips, see [High CPU usage](high-cpu-usage.md) and [High JVM memory pressure](high-jvm-memory-pressure.md). + +**Workaround for semantic_text field indexing issues:** + +For semantic_text field indexing issues (versions 8.19+ and 9.1+), try reducing the batch size of documents being indexed. + +**Resolution for semantic_text field issues:** + +When working with semantic_text fields in versions 8.19+ and 9.1+: + +1. First, try reducing the batch size of documents in your indexing requests. +2. If reducing batch size doesn't resolve the issue, then consider scaling up your machine resources. +3. A last resort option is to adjust the indexing_pressure.memory.coordinating.limit cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur. This adjustment should only be considered after exhausting other options, as setting this value too high may risk Out of Memory (OOM) errors in your cluster. A cluster restart is required for this change to take effect. \ No newline at end of file From 0fb79ed394bf0e2d6024a394311ee8fc3ee340e7 Mon Sep 17 00:00:00 2001 From: Michail Romaios Date: Mon, 11 Aug 2025 17:32:25 +0200 Subject: [PATCH 2/8] adjust text --- troubleshoot/elasticsearch/rejected-requests.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/troubleshoot/elasticsearch/rejected-requests.md b/troubleshoot/elasticsearch/rejected-requests.md index fc8b84e825..597bf3eea2 100644 --- a/troubleshoot/elasticsearch/rejected-requests.md +++ b/troubleshoot/elasticsearch/rejected-requests.md @@ -80,11 +80,7 @@ See [this video](https://www.youtube.com/watch?v=QuV8QqSfc0c) for a walkthrough If {{es}} regularly rejects requests and other tasks, your cluster likely has high CPU usage or high JVM memory pressure. For tips, see [High CPU usage](high-cpu-usage.md) and [High JVM memory pressure](high-jvm-memory-pressure.md). -**Workaround for semantic_text field indexing issues:** - -For semantic_text field indexing issues (versions 8.19+ and 9.1+), try reducing the batch size of documents being indexed. - -**Resolution for semantic_text field issues:** +**Fix for semantic_text field issues:** When working with semantic_text fields in versions 8.19+ and 9.1+: From 88d61fb6d7fd61bd554228854b87fef75a013bd9 Mon Sep 17 00:00:00 2001 From: Michail Romaios Date: Mon, 11 Aug 2025 17:39:38 +0200 Subject: [PATCH 3/8] format --- troubleshoot/elasticsearch/rejected-requests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/troubleshoot/elasticsearch/rejected-requests.md b/troubleshoot/elasticsearch/rejected-requests.md index 597bf3eea2..c193ef9fe1 100644 --- a/troubleshoot/elasticsearch/rejected-requests.md +++ b/troubleshoot/elasticsearch/rejected-requests.md @@ -86,4 +86,4 @@ When working with semantic_text fields in versions 8.19+ and 9.1+: 1. First, try reducing the batch size of documents in your indexing requests. 2. If reducing batch size doesn't resolve the issue, then consider scaling up your machine resources. -3. A last resort option is to adjust the indexing_pressure.memory.coordinating.limit cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur. This adjustment should only be considered after exhausting other options, as setting this value too high may risk Out of Memory (OOM) errors in your cluster. A cluster restart is required for this change to take effect. \ No newline at end of file +3. A last resort option is to adjust the `indexing_pressure.memory.coordinating.limit` cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur. This adjustment should only be considered after exhausting other options, as setting this value too high may risk Out of Memory (OOM) errors in your cluster. A cluster restart is required for this change to take effect. \ No newline at end of file From 0206b4c802e33cf050dba4289bd8ca17ea8ca683 Mon Sep 17 00:00:00 2001 From: Michail Romaios <84708293+mromaios@users.noreply.github.com> Date: Mon, 11 Aug 2025 17:43:56 +0200 Subject: [PATCH 4/8] Update troubleshoot/elasticsearch/rejected-requests.md Co-authored-by: Kathleen DeRusso --- troubleshoot/elasticsearch/rejected-requests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/troubleshoot/elasticsearch/rejected-requests.md b/troubleshoot/elasticsearch/rejected-requests.md index c193ef9fe1..015c010cd5 100644 --- a/troubleshoot/elasticsearch/rejected-requests.md +++ b/troubleshoot/elasticsearch/rejected-requests.md @@ -84,6 +84,6 @@ If {{es}} regularly rejects requests and other tasks, your cluster likely has hi When working with semantic_text fields in versions 8.19+ and 9.1+: -1. First, try reducing the batch size of documents in your indexing requests. +1. Reduce the batch size of documents in your indexing requests. 2. If reducing batch size doesn't resolve the issue, then consider scaling up your machine resources. 3. A last resort option is to adjust the `indexing_pressure.memory.coordinating.limit` cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur. This adjustment should only be considered after exhausting other options, as setting this value too high may risk Out of Memory (OOM) errors in your cluster. A cluster restart is required for this change to take effect. \ No newline at end of file From 951e2b43b231d859cea52a8fe0d1ef7203421ec7 Mon Sep 17 00:00:00 2001 From: Michail Romaios Date: Mon, 11 Aug 2025 18:04:01 +0200 Subject: [PATCH 5/8] trying out markdown applies --- troubleshoot/elasticsearch/rejected-requests.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/troubleshoot/elasticsearch/rejected-requests.md b/troubleshoot/elasticsearch/rejected-requests.md index 015c010cd5..01be4844d1 100644 --- a/troubleshoot/elasticsearch/rejected-requests.md +++ b/troubleshoot/elasticsearch/rejected-requests.md @@ -67,9 +67,8 @@ These stats are cumulative from node startup. Indexing pressure rejections appear as an `EsRejectedExecutionException`, and indicate that they were rejected due to `combined_coordinating_and_primary`, `coordinating`, `primary`, or `replica`. -These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](elasticsearch://reference/elasticsearch/index-settings/index-modules.md).¹ - -¹ : Starting from versions 8.19+ and 9.1+, indexing to a semantic_text field can also cause 429 errors if the batch may otherwise incur an Out Of Memory (OOM) error. +These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](elasticsearch://reference/elasticsearch/index-settings/index-modules.md). +{applies_to}`stack: ga 9.1`Another cause of indexing pressure rejections might be the use of the [`semantic_text`](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text) field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out Of Memory (OOM) error. See [this video](https://www.youtube.com/watch?v=QuV8QqSfc0c) for a walkthrough of diagnosing indexing pressure rejections. @@ -80,10 +79,8 @@ See [this video](https://www.youtube.com/watch?v=QuV8QqSfc0c) for a walkthrough If {{es}} regularly rejects requests and other tasks, your cluster likely has high CPU usage or high JVM memory pressure. For tips, see [High CPU usage](high-cpu-usage.md) and [High JVM memory pressure](high-jvm-memory-pressure.md). -**Fix for semantic_text field issues:** - -When working with semantic_text fields in versions 8.19+ and 9.1+: +{applies_to}`stack: ga 9.1`**Fix for semantic_text field issues:** 1. Reduce the batch size of documents in your indexing requests. -2. If reducing batch size doesn't resolve the issue, then consider scaling up your machine resources. -3. A last resort option is to adjust the `indexing_pressure.memory.coordinating.limit` cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur. This adjustment should only be considered after exhausting other options, as setting this value too high may risk Out of Memory (OOM) errors in your cluster. A cluster restart is required for this change to take effect. \ No newline at end of file +2. If reducing batch size doesn't resolve the issue, then consider scaling up your machine resources. +3. {applies_to}`serverless: unavailable`A last resort option is to adjust the `indexing_pressure.memory.coordinating.limit` cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur. \ No newline at end of file From e0b1d085586494918244340e415fd6bb0ca4effd Mon Sep 17 00:00:00 2001 From: Michail Romaios Date: Mon, 11 Aug 2025 18:27:16 +0200 Subject: [PATCH 6/8] further markdown exp --- .../elasticsearch/rejected-requests.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/troubleshoot/elasticsearch/rejected-requests.md b/troubleshoot/elasticsearch/rejected-requests.md index 01be4844d1..8127dff6bf 100644 --- a/troubleshoot/elasticsearch/rejected-requests.md +++ b/troubleshoot/elasticsearch/rejected-requests.md @@ -67,20 +67,27 @@ These stats are cumulative from node startup. Indexing pressure rejections appear as an `EsRejectedExecutionException`, and indicate that they were rejected due to `combined_coordinating_and_primary`, `coordinating`, `primary`, or `replica`. -These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](elasticsearch://reference/elasticsearch/index-settings/index-modules.md). -{applies_to}`stack: ga 9.1`Another cause of indexing pressure rejections might be the use of the [`semantic_text`](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text) field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out Of Memory (OOM) error. +These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](elasticsearch://reference/elasticsearch/index-settings/index-modules.md). +{applies_to}`stack: ga 9.1`{applies_to}`serverless: ga`Another cause of indexing pressure rejections might be the use of the [`semantic_text`](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text) field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out Of Memory (OOM) error. See [this video](https://www.youtube.com/watch?v=QuV8QqSfc0c) for a walkthrough of diagnosing indexing pressure rejections. ## Prevent rejected requests [prevent-rejected-requests] -**Fix high CPU and memory usage** +### Fix high CPU and memory usage [fix-high-cpu-memory-usage] If {{es}} regularly rejects requests and other tasks, your cluster likely has high CPU usage or high JVM memory pressure. For tips, see [High CPU usage](high-cpu-usage.md) and [High JVM memory pressure](high-jvm-memory-pressure.md). -{applies_to}`stack: ga 9.1`**Fix for semantic_text field issues:** +### Fix for `semantic_text` ingestion issues [fix-semantic-text-ingestion-issues] +```{applies_to} +stack: ga 9.1 +serverless: ga +``` 1. Reduce the batch size of documents in your indexing requests. -2. If reducing batch size doesn't resolve the issue, then consider scaling up your machine resources. -3. {applies_to}`serverless: unavailable`A last resort option is to adjust the `indexing_pressure.memory.coordinating.limit` cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur. \ No newline at end of file +2. If reducing batch size doesn't resolve the issue, then consider scaling up your machine resources. +{applies_to}`serverless: unavailable`3. A last resort option is to adjust the `indexing_pressure.memory.coordinating.limit` cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur. +::::{warning} +This adjustment should only be considered after exhausting other options, as setting this value too high may risk Out of Memory (OOM) errors in your cluster. A cluster restart is required for this change to take effect. +:::: \ No newline at end of file From 97d4cb2c000c1ec41c5c5f10bdb51cf016653987 Mon Sep 17 00:00:00 2001 From: Michail Romaios <84708293+mromaios@users.noreply.github.com> Date: Tue, 12 Aug 2025 10:33:51 +0200 Subject: [PATCH 7/8] Update troubleshoot/elasticsearch/rejected-requests.md Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> --- troubleshoot/elasticsearch/rejected-requests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/troubleshoot/elasticsearch/rejected-requests.md b/troubleshoot/elasticsearch/rejected-requests.md index 8127dff6bf..b5129eb354 100644 --- a/troubleshoot/elasticsearch/rejected-requests.md +++ b/troubleshoot/elasticsearch/rejected-requests.md @@ -68,7 +68,7 @@ These stats are cumulative from node startup. Indexing pressure rejections appear as an `EsRejectedExecutionException`, and indicate that they were rejected due to `combined_coordinating_and_primary`, `coordinating`, `primary`, or `replica`. These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](elasticsearch://reference/elasticsearch/index-settings/index-modules.md). -{applies_to}`stack: ga 9.1`{applies_to}`serverless: ga`Another cause of indexing pressure rejections might be the use of the [`semantic_text`](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text) field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out Of Memory (OOM) error. +{applies_to}`stack: ga 9.1`{applies_to}`serverless: ga`Another cause of indexing pressure rejections might be the use of the [`semantic_text`](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text) field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out of Memory (OOM) error. See [this video](https://www.youtube.com/watch?v=QuV8QqSfc0c) for a walkthrough of diagnosing indexing pressure rejections. From 241c1faa3f9f6bb81aec630868de069771762eaf Mon Sep 17 00:00:00 2001 From: Michail Romaios Date: Tue, 12 Aug 2025 13:56:06 +0200 Subject: [PATCH 8/8] fix formatting, address pr comments --- troubleshoot/elasticsearch/rejected-requests.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/troubleshoot/elasticsearch/rejected-requests.md b/troubleshoot/elasticsearch/rejected-requests.md index 8127dff6bf..a0b16bad78 100644 --- a/troubleshoot/elasticsearch/rejected-requests.md +++ b/troubleshoot/elasticsearch/rejected-requests.md @@ -68,7 +68,11 @@ These stats are cumulative from node startup. Indexing pressure rejections appear as an `EsRejectedExecutionException`, and indicate that they were rejected due to `combined_coordinating_and_primary`, `coordinating`, `primary`, or `replica`. These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](elasticsearch://reference/elasticsearch/index-settings/index-modules.md). -{applies_to}`stack: ga 9.1`{applies_to}`serverless: ga`Another cause of indexing pressure rejections might be the use of the [`semantic_text`](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text) field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out Of Memory (OOM) error. + +::::{note} +{applies_to}`stack: ga 9.1`{applies_to}`serverless: ga` +Another cause of indexing pressure rejections might be the use of the [`semantic_text`](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text) field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out Of Memory (OOM) error. +:::: See [this video](https://www.youtube.com/watch?v=QuV8QqSfc0c) for a walkthrough of diagnosing indexing pressure rejections. @@ -84,10 +88,14 @@ If {{es}} regularly rejects requests and other tasks, your cluster likely has hi stack: ga 9.1 serverless: ga ``` +When bulk indexing documents with the `semantic_text` field type, you may encounter rejections due to high memory usage during inference processing. +These rejections will appear as an `InferenceException` in your cluster logs. + +**To resolve this issue:** 1. Reduce the batch size of documents in your indexing requests. 2. If reducing batch size doesn't resolve the issue, then consider scaling up your machine resources. -{applies_to}`serverless: unavailable`3. A last resort option is to adjust the `indexing_pressure.memory.coordinating.limit` cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur. +3. {applies_to}`serverless: unavailable` A last resort option is to adjust the `indexing_pressure.memory.coordinating.limit` cluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur. ::::{warning} This adjustment should only be considered after exhausting other options, as setting this value too high may risk Out of Memory (OOM) errors in your cluster. A cluster restart is required for this change to take effect. :::: \ No newline at end of file