From 82598b52692f21666c3544bf9f065f2f24e69937 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri Benedetti Date: Wed, 15 Oct 2025 18:53:29 +0200 Subject: [PATCH 1/2] Add batching config docs --- .../config/default-config-standalone.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/reference/edot-collector/config/default-config-standalone.md b/docs/reference/edot-collector/config/default-config-standalone.md index 5d5f7b774ef..477a573565d 100644 --- a/docs/reference/edot-collector/config/default-config-standalone.md +++ b/docs/reference/edot-collector/config/default-config-standalone.md @@ -102,6 +102,33 @@ Data from OTel SDKs is piped through the [`OTLP`] receiver directly to the OTLP With the {{motlp}}, there is no need to configure any Elastic-specific components, such as the [`elasticinframetrics`] and [`elasticapm`] processors, the [`elasticapm`] connector, or the [`elasticsearch`] exporter. Edge setup and configuration can be fully vendor agnostic. +### Batching configuration for contrib OpenTelemetry Collector + +Contrib OpenTelemetry collectors should use the following batching configuration when sending data to the {{motlp}}. This configuration is already included in EDOT and optimizes data transfer to the managed endpoint: + +```yaml +otlp/ingest: + endpoint: + headers: + Authorization: ApiKey + sending_queue: + enabled: true + sizer: bytes + queue_size: 50000000 # 50MB uncompressed + block_on_overflow: true + batch: + flush_interval: 1s + min_size: 1_000_000 # 1MB uncompressed + max_size: 4_000_000 # 4MB uncompressed +``` + +The batching configuration ensures efficient data transfer by: + +- Using a 50MB queue to buffer data before sending. +- Flushing batches at least every second. +- Sending batches between 1MB and 4MB in size. +- Blocking when the queue is full to prevent data loss. + ## Gateway mode In Gateway mode, the Collector ingests data from other Collectors running in Agent mode and forwards it to Elastic. From d9495406894e7973e0d3e936f5ad1651e038540e Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri Benedetti Date: Thu, 16 Oct 2025 13:08:24 +0200 Subject: [PATCH 2/2] Tech review edits --- .../config/default-config-standalone.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/reference/edot-collector/config/default-config-standalone.md b/docs/reference/edot-collector/config/default-config-standalone.md index 477a573565d..2bf272ace8c 100644 --- a/docs/reference/edot-collector/config/default-config-standalone.md +++ b/docs/reference/edot-collector/config/default-config-standalone.md @@ -13,7 +13,7 @@ products: - id: edot-collector --- -# Default configuration of the EDOT Collector (Standalone) +# Default configuration of the EDOT Collector (standalone) The default configuration of the {{edot}} (EDOT) Collector includes pipelines for the collection of logs, host metrics, and data from OpenTelemetry SDKs. @@ -65,7 +65,7 @@ Data is exported directly to {{es}} using the [`elasticsearch`] exporter in `OTe The application pipeline in the EDOT Collector receives data from OTel SDKs through the [`OTLP`] receiver. While logs and metrics are exported verbatim into {{es}}, traces require two additional components. -{applies_to}`edot_collector: ga 9.2` The [`elasticapm`] processor enriches trace data with additional attributes that improve the user experience in the Elastic Observability UIs. In addition, the [`elasticapm`] connector generates pre-aggregated APM metrics from tracing data. +{applies_to}`edot_collector: ga 9.2` The [`elasticapm`] processor enriches trace data with additional attributes that improve the user experience in the {{product.observability}} UIs. In addition, the [`elasticapm`] connector generates pre-aggregated APM metrics from tracing data. Application-related OTel data is ingested into {{es}} in OTel-native format using the [`elasticsearch`] exporter. @@ -104,11 +104,11 @@ With the {{motlp}}, there is no need to configure any Elastic-specific component ### Batching configuration for contrib OpenTelemetry Collector -Contrib OpenTelemetry collectors should use the following batching configuration when sending data to the {{motlp}}. This configuration is already included in EDOT and optimizes data transfer to the managed endpoint: +When using contrib or upstream OpenTelemetry collectors, the following batching configuration is recommended when sending data to the {{motlp}}: ```yaml otlp/ingest: - endpoint: + endpoint: headers: Authorization: ApiKey sending_queue: @@ -122,12 +122,11 @@ otlp/ingest: max_size: 4_000_000 # 4MB uncompressed ``` -The batching configuration ensures efficient data transfer by: +The previous configuration leverages an in-memory queue and optimized batching defaults to improve throughput, minimize data loss, and maintain low end-to-end latency. -- Using a 50MB queue to buffer data before sending. -- Flushing batches at least every second. -- Sending batches between 1MB and 4MB in size. -- Blocking when the queue is full to prevent data loss. +:::{note} +The previous configuration is already included in the {{edot}} Collector. +::: ## Gateway mode