diff --git a/docs/reference/edot-collector/config/default-config-standalone.md b/docs/reference/edot-collector/config/default-config-standalone.md new file mode 100644 index 00000000000..2bf272ace8c --- /dev/null +++ b/docs/reference/edot-collector/config/default-config-standalone.md @@ -0,0 +1,370 @@ +--- +navigation_title: Default config (Standalone) +description: Default configuration of the EDOT Collector in standalone mode. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Default configuration of the EDOT Collector (standalone) + +The default configuration of the {{edot}} (EDOT) Collector includes pipelines for the collection of logs, host metrics, and data from OpenTelemetry SDKs. + +The EDOT Collector can run in [Agent](https://opentelemetry.io/docs/collector/deployment/agent/) or [Gateway](https://opentelemetry.io/docs/collector/deployment/gateway/) mode: + +- Agent mode: The EDOT Collector ingests data from infrastructure and SDKs and forwards it to Elastic or to another collector running in Gateway mode. +- Gateway mode: The EDOT Collector ingests data from other collectors running in Agent mode and forwards it to Elastic. + +## Agent mode + +The following sample config files for Agent mode are available: + +| Use Cases | Direct ingestion into {{es}} | Managed OTLP Endpoint | +|---|---|---| +| Platform logs | [Logs - ES] | [Logs - OTLP] | +| Platform logs and host metrics | [Logs | Metrics - ES] | [Logs | Metrics - OTLP] | +| Platform logs, host metrics,
and application telemetry | [Logs | Metrics | App - ES]
(*default*) | [Logs | Metrics | App - OTLP]
(*default*) | + +Use the previous example configurations as a reference when configuring your contrib Collector or customizing your EDOT Collector configuration. + +The following sections describe the default pipelines by use cases. + +### Direct ingestion into Elasticsearch + +For self-managed and {{ech}} stack deployment use cases, ingest OpenTelemetry data from the EDOT Collector directly into {{es}} using the [`elasticsearch`] exporter. + +Learn more about the configuration options for the `elasticsearch` exporter in the [corresponding documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/README.md#configuration-options). + +The `elasticsearch` exporter comes with two relevant data ingestion modes: + +- `ecs`: Writes data in backwards compatible Elastic Common Schema (ECS) format. Original attribute names and semantics might be lost during translation. +- `otel`: OTel attribute names and semantics are preserved. + +The goal of EDOT is to preserve OTel data formats and semantics as much as possible, so `otel` is the default mode for the EDOT Collector. Some use cases might require data to be exported in ECS format for backwards compatibility. + +#### Logs collection pipeline + +For logs collection, the default configuration uses the [`filelog`] receiver to read log entries from files. In addition, the [`resourcedetection`] processor enriches the log entries with metadata about the corresponding host and operating system. + +:::{note} +The `from_context: client_metadata` option in the `resource` processor only applies to transport-level metadata. It cannot extract custom application attributes. + +To propagate such values into your telemetry, set them explicitly in your application code using EDOT SDK instrumentation. For more information, refer to [EDOT Collector doesn’t propagate client metadata](docs-content://troubleshoot/ingest/opentelemetry/edot-collector/metadata.md). +::: + +Data is exported directly to {{es}} using the [`elasticsearch`] exporter in `OTel-native` mode. + +#### Application and traces collection pipeline + +The application pipeline in the EDOT Collector receives data from OTel SDKs through the [`OTLP`] receiver. While logs and metrics are exported verbatim into {{es}}, traces require two additional components. + +{applies_to}`edot_collector: ga 9.2` The [`elasticapm`] processor enriches trace data with additional attributes that improve the user experience in the {{product.observability}} UIs. In addition, the [`elasticapm`] connector generates pre-aggregated APM metrics from tracing data. + +Application-related OTel data is ingested into {{es}} in OTel-native format using the [`elasticsearch`] exporter. + +:::{note} +Both the `elasticapm` processor and the `elasticapm` connector are required for Elastic APM UIs to work properly. As they aren't included in the OpenTelemetry [Collector Contrib repository](https://github.com/open-telemetry/opentelemetry-collector-contrib), you can: + +* Use the EDOT Collector with the available configuration to ingest data into {{es}}. +* [Build a custom, EDOT-like Collector](/reference/edot-collector/custom-collector.md) for ingesting data into {{es}}. +* Use Elastic's [managed OTLP endpoint](docs-content://solutions/observability/get-started/opentelemetry/quickstart/serverless/index.md) that does the enrichment for you. +::: + +#### Host metrics collection pipeline + +The host metrics pipeline uses the [`hostmetrics`] receiver to collect `disk`, `filesystem`, `cpu`, `memory`, `process` and `network` metrics for the corresponding host. + +For backwards compatibility, host metrics are translated into ECS-compatible system metrics using the [`elasticinframetrics`] processor. Finally, metrics are ingested in `ecs` format through the [`elasticsearch`] exporter. + +The [`resourcedetection`] processor enriches the metrics with meta information about the corresponding host and operating system. The [`attributes`] and [`resource`] processor are used to set some fields for proper routing of the ECS-based system metrics data into corresponding {{es}} data streams. + +::::{important} +:::{include} ../_snippets/process-config.md +::: +:::: + +### Using the Managed OTLP Endpoint + +When ingesting OTel data through the [{{motlp}}](opentelemetry://reference/motlp.md), all the enrichment that is required for an optimal experience in the Elastic solutions happens at the endpoint level and is transparent to users. + +The Collector configuration for all use cases that involve the {{motlp}} is only concerned with local data collection and context enrichment. + +Platform logs are scraped with the [`filelog`] receiver, host metrics are collected through the [`hostmetrics`] receiver and both signals are enriched with meta information through the [`resourcedetection`] processor. + +Data from OTel SDKs is piped through the [`OTLP`] receiver directly to the OTLP exporter that sends data for all signals to the {{motlp}}. + +With the {{motlp}}, there is no need to configure any Elastic-specific components, such as the [`elasticinframetrics`] and [`elasticapm`] processors, the [`elasticapm`] connector, or the [`elasticsearch`] exporter. Edge setup and configuration can be fully vendor agnostic. + +### Batching configuration for contrib OpenTelemetry Collector + +When using contrib or upstream OpenTelemetry collectors, the following batching configuration is recommended when sending data to the {{motlp}}: + +```yaml +otlp/ingest: + endpoint: + headers: + Authorization: ApiKey + sending_queue: + enabled: true + sizer: bytes + queue_size: 50000000 # 50MB uncompressed + block_on_overflow: true + batch: + flush_interval: 1s + min_size: 1_000_000 # 1MB uncompressed + max_size: 4_000_000 # 4MB uncompressed +``` + +The previous configuration leverages an in-memory queue and optimized batching defaults to improve throughput, minimize data loss, and maintain low end-to-end latency. + +:::{note} +The previous configuration is already included in the {{edot}} Collector. +::: + +## Gateway mode + +In Gateway mode, the Collector ingests data from other Collectors running in Agent mode and forwards it to Elastic. + +### Example configuration + +The following example configuration files are available for the Gateway mode: + +:::::{tab-set} + +::::{tab-item} 9.x +% start:edot-gateway-9x-table +| Version | Configuration | +|---------|----------------| +| 9.1 | [Gateway mode](https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v9.1.5/internal/pkg/otel/samples/linux/gateway.yml) | +| 9.0 | [Gateway mode](https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v9.0.8/internal/pkg/otel/samples/linux/gateway.yml) | +% end:edot-gateway-9x-table +:::: + +::::{tab-item} 8.x +% start:edot-gateway-8x-table +| Version | Configuration | +|---------|----------------| +| 8.19 | [Gateway mode](https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v8.19.5/internal/pkg/otel/samples/linux/gateway.yml) | +| 8.18 | [Gateway mode](https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v8.18.8/internal/pkg/otel/samples/linux/gateway.yml) | +| 8.17 | [Gateway mode](https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v8.17.10/internal/pkg/otel/samples/linux/gateway.yml) | +% end:edot-gateway-8x-table +:::: +::::: + +Use the previous example configuration as a reference when configuring your Gateway Collector or customizing your EDOT Collector configuration. + +### Data collection and processing + +The EDOT Collector in Gateway mode collects data from other Collectors using the OTLP protocol. By default, the sample Gateway configuration listens on port `4317` for gRPC and port `4318` for HTTP. + +```yaml +receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 # Listen on all interfaces + http: + endpoint: 0.0.0.0:4318 # Listen on all interfaces +``` + +The routing connector splits infrastructure metrics from other metrics and routes them to the appropriate Elastic Common Schema pipelines. Other metrics are exported in OTel-native format through the [`elasticsearch`] exporter. + +```yaml +connectors: + routing: + default_pipelines: [metrics/otel] + error_mode: ignore + table: + - context: metric + statement: route() where IsMatch(instrumentation_scope.name, "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/*") + pipelines: [metrics/infra/ecs, metrics/otel] + elasticapm: {} +``` + +### Data processing and transformation + +The Gateway configuration includes several processors to transform and optimize the collected data: + +```yaml +processors: + elasticinframetrics: + add_system_metrics: true + drop_original: true + attributes/dataset: + actions: + - key: event.dataset + from_attribute: data_stream.dataset + action: upsert + resource/process: + attributes: + - key: process.executable.name + action: delete + - key: process.executable.path + action: delete + batch: + send_batch_size: 1000 + timeout: 1s + send_batch_max_size: 1500 + batch/metrics: + send_batch_max_size: 0 # Prevents splitting metrics requests + timeout: 1s + elasticapm: {} +``` + +:::{note} +:applies_to: edot_collector: ga 9.2 +The `elasticapm` processor replaces the deprecated `elastictrace` processor. +::: + +### Data export + +The Gateway exports data to Elasticsearch in two formats: + +- OTel-native format using the `elasticsearch/otel` exporter. +- Elastic Common Schema (ECS) format using the `elasticsearch/ecs` exporter. + +```yaml +exporters: + elasticsearch/otel: + endpoints: + - ${ELASTIC_ENDPOINT} + api_key: ${ELASTIC_API_KEY} + mapping: + mode: otel + elasticsearch/ecs: + endpoints: + - ${ELASTIC_ENDPOINT} + api_key: ${ELASTIC_API_KEY} + mapping: + mode: ecs +``` + +### Pipeline configuration + +The service section defines separate pipelines for different telemetry types: + +- Metrics pipelines for infrastructure and OTel metrics +- Logs pipeline +- Traces pipeline +- Aggregated OTel metrics pipeline + +Each pipeline connects specific receivers, processors, and exporters to handle different data types appropriately. + +## Central configuration + +The EDOT Collector can be configured to use [APM Agent Central Configuration](docs-content://solutions/observability/apm/apm-agent-central-configuration.md). Refer to [Central configuration docs](opentelemetry://reference/central-configuration.md) for more details. + +To activate the central configuration feature, add the [`apmconfig`](https://github.com/elastic/opentelemetry-collector-components/blob/main/extension/apmconfigextension/README.md). For example: + +:::{include} ../_snippets/edot-collector-auth.md +::: + +Create an API Key following [these instructions](docs-content://deploy-manage/api-keys/elasticsearch-api-keys.md). The API key must have `config_agent:read` permissions and resources set to `-`. + +## Secure connection + +To secure the connection between the EDOT Collector and Elastic, you can use TLS or mutual TLS, as well as the `apikeyauth` extension. + +### TLS configuration + +You can turn on TLS or mutual TLS to encrypt data in transit between EDOT SDKs and the extension. Refer to [OpenTelemetry TLS server configuration](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#server-configuration) for more details. + +For example: + +```yaml +extensions: + apmconfig: + opamp: + protocols: + http: + endpoint: ":4320" + tls: + cert_file: server.crt + key_file: server.key + ... +``` + +### Authentication settings + +In addition to TLS, you can configure authentication to ensure that only authorized agents can communicate with the extension and retrieve their corresponding remote configurations. + +The `apmconfig` extension supports any configauth authenticator. Use the `apikeyauth` extension to authenticate with Elasticsearch API keys: + +```yaml +extensions: + apikeyauth: + endpoint: "" + application_privileges: + - application: "apm" + privileges: + - "config_agent:read" + resources: + - "-" + apmconfig: + opamp: + protocols: + http: + auth: + authenticator: apikeyauth + ... +``` + +Create an API key with the minimum required application permissions through {{kib}} under **Observability** → **Applications** → **Settings** → **Agent Keys**, or by using the Elasticsearch Security API: + +::::{dropdown} Example JSON payload +```json +POST /_security/api_key +{ + "name": "apmconfig-opamp-test-sdk", + "metadata": { + "application": "apm" + }, + "role_descriptors": { + "apm": { + "cluster": [], + "indices": [], + "applications": [ + { + "application": "apm", + "privileges": [ + "config_agent:read" + ], + "resources": [ + "*" + ] + } + ], + "run_as": [], + "metadata": {} + } + } +} +``` +:::: + +The server expects incoming HTTP requests to include an API key with sufficient privileges, using the following header format: `Authorization: ApiKey `. + +[`attributes`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor +[`filelog`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver +[`hostmetrics`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver +[`elasticsearch`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter +[`elasticinframetrics`]: https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/elasticinframetricsprocessor +[`elasticapm`]: https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/elasticapmprocessor +[`elasticapm`]: https://github.com/elastic/opentelemetry-collector-components/tree/main/connector/elasticapmconnector +[`resource`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourceprocessor +[`resourcedetection`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor +[`OTLP`]: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver +[Logs - ES]: https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/internal/pkg/otel/samples/linux/platformlogs.yml +[Logs - OTLP]: https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/internal/pkg/otel/samples/linux/managed_otlp/platformlogs.yml +[Logs | Metrics - ES]: https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/internal/pkg/otel/samples/linux/platformlogs_hostmetrics.yml +[Logs | Metrics - OTLP]: https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/internal/pkg/otel/samples/linux/managed_otlp/platformlogs_hostmetrics.yml +[Logs | Metrics | App - ES]: https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/internal/pkg/otel/samples/linux/logs_metrics_traces.yml +[Logs | Metrics | App - OTLP]: https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/internal/pkg/otel/samples/linux/managed_otlp/logs_metrics_traces.yml +[Gateway mode]: https://raw.githubusercontent.com/elastic/elastic-agent/refs/heads/main/internal/pkg/otel/samples/linux/gateway.yml