From f32c745387437c9b3567d6e2a20d2c38ea2ff473 Mon Sep 17 00:00:00 2001 From: Francesco Gualazzi Date: Wed, 15 Oct 2025 15:21:25 +0200 Subject: [PATCH 1/8] Update Elasticsearch exporter documentation Clarified configuration options for sending queue and batcher settings in the Elasticsearch exporter documentation. Closes #10584 --- .../components/elasticsearchexporter.md | 36 +++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/docs/reference/edot-collector/components/elasticsearchexporter.md b/docs/reference/edot-collector/components/elasticsearchexporter.md index c25282d4f56..5eca9e78282 100644 --- a/docs/reference/edot-collector/components/elasticsearchexporter.md +++ b/docs/reference/edot-collector/components/elasticsearchexporter.md @@ -135,11 +135,43 @@ The exporter supports both internal batching and OpenTelemetry's standard `sendi ### Internal batching (default) -By default, the exporter performs its own buffering and batching, as configured through the `flush` setting, unless the `sending_queue` and `batcher` settings are defined. +By default, the exporter performs its own buffering and batching, as configured through the `flush` setting, unless the `sending_queue` and/or `batcher` settings are defined. ### Using sending queue -The Elasticsearch exporter supports the `sending_queue` setting, which supports both queueing and batching. However, the sending queue is currently deactivated by default. You can turn on the sending queue by setting `sending_queue` to true. Batching support in sending queue is also deactivated by default and can be turned on by defining `sending_queue::batch`. For example: +```{applies_to} +stack: 9.0 9.1 +``` + +The sending queue can be enabled and configured with the `batcher` section, using [common `batcher` settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/internal/queue_sender.go). + +- `batcher`: + - `enabled` (default=unset): Enable batching of requests into 1 or more bulk requests. On a batcher flush, it is possible for a batched request to be translated to more than 1 bulk request due to `flush::bytes`. + - `sizer` (default=items): Unit of `min_size` and `max_size`. Currently supports only "items", in the future will also support "bytes". + - `min_size` (default=5000): Minimum batch size to be exported to Elasticsearch, measured in units according to `batcher::sizer`. + - `max_size` (default=0): Maximum batch size to be exported to Elasticsearch, measured in units according to `batcher::sizer`. To limit bulk request size, configure `flush::bytes` instead. :warning: It is recommended to keep `max_size` as 0 as a non-zero value may lead to broken metrics grouping and indexing rejections. + - `flush_timeout` (default=10s): Maximum time of the oldest item spent inside the batcher buffer, aka "max age of batcher buffer". A batcher flush will happen regardless of the size of content in batcher buffer. + +For example: + +```yaml subs=true +exporters: + elasticsearch: + endpoint: https://elasticsearch:9200 + batcher: + enabled: true + min_size: 1000 + max_size: 10000 + flush_timeout: 5s +``` + +```{applies to} +stack: 9.2 +``` + +The Elasticsearch exporter supports the `sending_queue` setting, which supports both queueing and batching. +However, the sending queue is currently deactivated by default. +You can turn on the sending queue by setting `sending_queue` to true. Batching support in sending queue is also deactivated by default and can be turned on by defining `sending_queue::batch`. For example: ```yaml subs=true exporters: From fd6fd34db118cfda765fb38b4d4757ac1996f494 Mon Sep 17 00:00:00 2001 From: Francesco Gualazzi Date: Wed, 15 Oct 2025 16:25:22 +0200 Subject: [PATCH 2/8] Update docs/reference/edot-collector/components/elasticsearchexporter.md Co-authored-by: Carson Ip --- .../edot-collector/components/elasticsearchexporter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/edot-collector/components/elasticsearchexporter.md b/docs/reference/edot-collector/components/elasticsearchexporter.md index 5eca9e78282..a30166eddfc 100644 --- a/docs/reference/edot-collector/components/elasticsearchexporter.md +++ b/docs/reference/edot-collector/components/elasticsearchexporter.md @@ -171,7 +171,7 @@ stack: 9.2 The Elasticsearch exporter supports the `sending_queue` setting, which supports both queueing and batching. However, the sending queue is currently deactivated by default. -You can turn on the sending queue by setting `sending_queue` to true. Batching support in sending queue is also deactivated by default and can be turned on by defining `sending_queue::batch`. For example: +You can turn on the sending queue by setting `sending_queue::enabled` to true. Batching support in sending queue is also deactivated by default and can be turned on by defining `sending_queue::batch`. For example: ```yaml subs=true exporters: From 1aaa8ca7b81197a641b62e031e020c3c50a7fd6a Mon Sep 17 00:00:00 2001 From: Francesco Gualazzi Date: Wed, 15 Oct 2025 19:00:52 +0200 Subject: [PATCH 3/8] Update docs/reference/edot-collector/components/elasticsearchexporter.md --- .../edot-collector/components/elasticsearchexporter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/edot-collector/components/elasticsearchexporter.md b/docs/reference/edot-collector/components/elasticsearchexporter.md index a30166eddfc..250ebadb43e 100644 --- a/docs/reference/edot-collector/components/elasticsearchexporter.md +++ b/docs/reference/edot-collector/components/elasticsearchexporter.md @@ -135,7 +135,7 @@ The exporter supports both internal batching and OpenTelemetry's standard `sendi ### Internal batching (default) -By default, the exporter performs its own buffering and batching, as configured through the `flush` setting, unless the `sending_queue` and/or `batcher` settings are defined. +By default, the exporter performs its own buffering and batching, as configured through the `flush` setting, unless the `sending_queue::batch` and/or `batcher` settings are defined. ### Using sending queue From 4dc3a02977e3025bd35c640f3aab1f3b8231a578 Mon Sep 17 00:00:00 2001 From: Francesco Gualazzi Date: Wed, 15 Oct 2025 21:44:47 +0200 Subject: [PATCH 4/8] fix applies_to syntax Co-authored-by: Fabrizio Ferri-Benedetti --- .../edot-collector/components/elasticsearchexporter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/edot-collector/components/elasticsearchexporter.md b/docs/reference/edot-collector/components/elasticsearchexporter.md index 250ebadb43e..03cdf8f6e6f 100644 --- a/docs/reference/edot-collector/components/elasticsearchexporter.md +++ b/docs/reference/edot-collector/components/elasticsearchexporter.md @@ -140,7 +140,7 @@ By default, the exporter performs its own buffering and batching, as configured ### Using sending queue ```{applies_to} -stack: 9.0 9.1 +stack: ga 9.0, deprecated 9.2 ``` The sending queue can be enabled and configured with the `batcher` section, using [common `batcher` settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/internal/queue_sender.go). From ff7cbe3d57d72f2816dff80f673694491f63d26b Mon Sep 17 00:00:00 2001 From: inge4pres Date: Wed, 15 Oct 2025 22:05:48 +0200 Subject: [PATCH 5/8] separate sending_queue and batching Signed-off-by: inge4pres --- .../components/elasticsearchexporter.md | 27 +++++++++++++------ 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/docs/reference/edot-collector/components/elasticsearchexporter.md b/docs/reference/edot-collector/components/elasticsearchexporter.md index 03cdf8f6e6f..ce1564e134c 100644 --- a/docs/reference/edot-collector/components/elasticsearchexporter.md +++ b/docs/reference/edot-collector/components/elasticsearchexporter.md @@ -131,19 +131,32 @@ The `elasticsearch.index` attribute is removed from the final document if it exi ## Performance and batching -The exporter supports both internal batching and OpenTelemetry's standard `sending_queue` configuration: +### Using sending queue + +The Elasticsearch exporter supports the `sending_queue` setting, which supports both queueing and batching. +However, the sending queue is currently deactivated by default. +You can turn on the sending queue by setting `sending_queue::enabled` to true. + +```yaml subs=true +exporters: + elasticsearch: + endpoint: https://elasticsearch:9200 + sending_queue: + enabled: true +``` ### Internal batching (default) By default, the exporter performs its own buffering and batching, as configured through the `flush` setting, unless the `sending_queue::batch` and/or `batcher` settings are defined. +In that case, batching is controlled by either of the two settings, depending on the version. -### Using sending queue +### Custom batching ```{applies_to} stack: ga 9.0, deprecated 9.2 ``` -The sending queue can be enabled and configured with the `batcher` section, using [common `batcher` settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/internal/queue_sender.go). +Batching can be enabled and configured with the `batcher` section, using [common `batcher` settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/internal/queue_sender.go). - `batcher`: - `enabled` (default=unset): Enable batching of requests into 1 or more bulk requests. On a batcher flush, it is possible for a batched request to be translated to more than 1 bulk request due to `flush::bytes`. @@ -165,13 +178,11 @@ exporters: flush_timeout: 5s ``` -```{applies to} -stack: 9.2 +```{applies_to} +stack: ga 9.2 ``` -The Elasticsearch exporter supports the `sending_queue` setting, which supports both queueing and batching. -However, the sending queue is currently deactivated by default. -You can turn on the sending queue by setting `sending_queue::enabled` to true. Batching support in sending queue is also deactivated by default and can be turned on by defining `sending_queue::batch`. For example: +Batching support in sending queue is also deactivated by default and can be turned on by defining `sending_queue::batch`. For example: ```yaml subs=true exporters: From 9c87b16942283fa03cbf5bc0c24af6bc46863eea Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Thu, 16 Oct 2025 12:48:28 +0200 Subject: [PATCH 6/8] Update elasticsearchexporter.md --- .../components/elasticsearchexporter.md | 53 ++++++++++--------- 1 file changed, 27 insertions(+), 26 deletions(-) diff --git a/docs/reference/edot-collector/components/elasticsearchexporter.md b/docs/reference/edot-collector/components/elasticsearchexporter.md index ce1564e134c..20c8502bd64 100644 --- a/docs/reference/edot-collector/components/elasticsearchexporter.md +++ b/docs/reference/edot-collector/components/elasticsearchexporter.md @@ -133,9 +133,9 @@ The `elasticsearch.index` attribute is removed from the final document if it exi ### Using sending queue -The Elasticsearch exporter supports the `sending_queue` setting, which supports both queueing and batching. -However, the sending queue is currently deactivated by default. -You can turn on the sending queue by setting `sending_queue::enabled` to true. +The {{es}} exporter includes the `sending_queue` setting, which supports both queueing and batching. The sending queue is deactivated by default. + +You can turn on the sending queue by setting `sending_queue::enabled` to `true`: ```yaml subs=true exporters: @@ -147,14 +147,32 @@ exporters: ### Internal batching (default) -By default, the exporter performs its own buffering and batching, as configured through the `flush` setting, unless the `sending_queue::batch` and/or `batcher` settings are defined. -In that case, batching is controlled by either of the two settings, depending on the version. +By default, the exporter performs its own buffering and batching, as configured through the `flush` setting, unless the `sending_queue::batch` or the `batcher` settings, or both, are defined. In that case, batching is controlled by either of the two settings, depending on the version. ### Custom batching -```{applies_to} -stack: ga 9.0, deprecated 9.2 +::::{applies-switch} + +:::{applies-item} stack: ga 9.2 +Batching support in sending queue is deactivated by default. To turn it on, define `sending_queue::batch`. + +For example: + +```yaml subs=true +exporters: + elasticsearch: + endpoint: https://elasticsearch:9200 + sending_queue: + enabled: true + batch: + enabled: true + min_size: 1000 + max_size: 10000 + timeout: 5s ``` +::: + +:::{applies-item} stack: ga 9.0, deprecated 9.2 Batching can be enabled and configured with the `batcher` section, using [common `batcher` settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/internal/queue_sender.go). @@ -177,25 +195,8 @@ exporters: max_size: 10000 flush_timeout: 5s ``` - -```{applies_to} -stack: ga 9.2 -``` - -Batching support in sending queue is also deactivated by default and can be turned on by defining `sending_queue::batch`. For example: - -```yaml subs=true -exporters: - elasticsearch: - endpoint: https://elasticsearch:9200 - sending_queue: - enabled: true - batch: - enabled: true - min_size: 1000 - max_size: 10000 - timeout: 5s -``` +::: +:::: ## Bulk indexing From ce5b86d66da8d50497b750ea125ae117c2df2ad9 Mon Sep 17 00:00:00 2001 From: Francesco Gualazzi Date: Thu, 16 Oct 2025 16:23:45 +0200 Subject: [PATCH 7/8] Apply suggestions from code review Co-authored-by: Carson Ip --- .../edot-collector/components/elasticsearchexporter.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/edot-collector/components/elasticsearchexporter.md b/docs/reference/edot-collector/components/elasticsearchexporter.md index 20c8502bd64..fd7c02c0ba6 100644 --- a/docs/reference/edot-collector/components/elasticsearchexporter.md +++ b/docs/reference/edot-collector/components/elasticsearchexporter.md @@ -133,7 +133,7 @@ The `elasticsearch.index` attribute is removed from the final document if it exi ### Using sending queue -The {{es}} exporter includes the `sending_queue` setting, which supports both queueing and batching. The sending queue is deactivated by default. +The {{es}} exporter supports the `sending_queue` setting, which supports both queueing and batching. The sending queue is deactivated by default. You can turn on the sending queue by setting `sending_queue::enabled` to `true`: @@ -147,14 +147,14 @@ exporters: ### Internal batching (default) -By default, the exporter performs its own buffering and batching, as configured through the `flush` setting, unless the `sending_queue::batch` or the `batcher` settings, or both, are defined. In that case, batching is controlled by either of the two settings, depending on the version. +By default, the exporter performs its own buffering and batching, as configured through the `flush` setting, unless the `sending_queue::batch` or the `batcher` settings are defined. In that case, batching is controlled by either of the two settings, depending on the version. ### Custom batching ::::{applies-switch} :::{applies-item} stack: ga 9.2 -Batching support in sending queue is deactivated by default. To turn it on, define `sending_queue::batch`. +Batching support in sending queue is deactivated by default. To turn it on, enable sending queue and define `sending_queue::batch`. For example: From acca0d7d7d4c0feeafbfc3b4ce0721c7f1f1b3b1 Mon Sep 17 00:00:00 2001 From: Francesco Gualazzi Date: Thu, 16 Oct 2025 18:13:18 +0200 Subject: [PATCH 8/8] Apply suggestion from @carsonip Co-authored-by: Carson Ip --- .../reference/edot-collector/components/elasticsearchexporter.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/edot-collector/components/elasticsearchexporter.md b/docs/reference/edot-collector/components/elasticsearchexporter.md index fd7c02c0ba6..2a9da23ee95 100644 --- a/docs/reference/edot-collector/components/elasticsearchexporter.md +++ b/docs/reference/edot-collector/components/elasticsearchexporter.md @@ -165,7 +165,6 @@ exporters: sending_queue: enabled: true batch: - enabled: true min_size: 1000 max_size: 10000 timeout: 5s