Add OTLP docs #3360
+126
−8
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| --- | ||
| navigation_title: "OTLP/HTTP endpoint" | ||
| applies_to: | ||
| stack: preview 9.2 | ||
theletterf marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| deployment: | ||
| self: | ||
| products: | ||
| - id: elasticsearch | ||
| --- | ||
|
|
||
| # OTLP/HTTP endpoint | ||
|
|
||
| :::{important} | ||
| The recommended way to send OTLP data for most use cases is through an OpenTelemetry Collector in [Gateway mode](elastic-agent://reference/edot-collector/config/default-config-standalone.md#gateway-mode) or, if you're on {{ecloud}}, directly to the [{{motlp}}](opentelemetry:/reference/motlp.md). Refer to [Best practices](#best-practices) for more details. | ||
| ::: | ||
|
|
||
| In addition to the ingestion of metrics data through the bulk API, | ||
| {{es}} offers an alternative way to ingest data through the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp). | ||
|
|
||
| The endpoint is available under `/_otlp/v1/metrics`. | ||
|
|
||
| Ingesting metrics data using the OTLP endpoint has the following advantages: | ||
|
|
||
| * Improved ingestion performance, especially if the data contains many resource attributes. | ||
| * Simplified index mapping: | ||
| there's no need to manually create data streams, index templates, or define dimensions and metrics. | ||
| Metrics are dynamically mapped using the metadata included in the OTLP requests. | ||
|
|
||
| :::{note} | ||
| {{es}} only supports [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp), | ||
| not [OTLP/gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc). | ||
| ::: | ||
|
|
||
| ## How to send data to the OTLP endpoint | ||
|
|
||
| To send data from an OpenTelemetry Collector to the {{es}} OTLP endpoint, | ||
| use the [`OTLP/HTTP` exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter). | ||
| This is an example configuration: | ||
|
|
||
| ```yaml | ||
| extensions: | ||
| basicauth/elasticsearch: | ||
| client_auth: | ||
| username: <user> | ||
| password: <password> | ||
| exporters: | ||
| otlphttp/elasticsearch-metrics: | ||
| endpoint: <es_endpoint>/_otlp | ||
| sending_queue: | ||
| enabled: true | ||
| sizer: bytes | ||
| queue_size: 50_000_000 # 50MB uncompressed | ||
| block_on_overflow: true | ||
| batch: | ||
| flush_timeout: 1s | ||
| min_size: 1_000_000 # 1MB uncompressed | ||
| max_size: 4_000_000 # 4MB uncompressed | ||
| auth: | ||
| authenticator: basicauth/elasticsearch | ||
| service: | ||
| extensions: [basicauth/elasticsearch] | ||
| pipelines: | ||
| metrics: | ||
| exporters: [otlphttp/elasticsearch-metrics] | ||
| receivers: ... | ||
| ``` | ||
|
|
||
| The supported options for `compression` are `gzip` (default value of the `OTLP/HTTP` exporter) and `none`. | ||
|
|
||
| % TODO we might actually also support snappy and zstd, test and update accordingly) | ||
|
|
||
| To track metrics in your custom application, | ||
| use the [OpenTelemetry language SDK](https://opentelemetry.io/docs/getting-started/dev/) of your choice. | ||
|
|
||
| :::{note} | ||
| Only `encoding: proto` is supported, which the `OTLP/HTTP` exporter uses by default. | ||
| ::: | ||
|
|
||
| ## Best practices | ||
|
|
||
| Don't send metrics from applications directly to the {{es}} OTLP endpoint, especially if there are many individual applications that periodically send a small amount of metrics. Instead, send data to an OpenTelemetry Collector first. This helps with handling many connections, and with creating bigger batches to improve ingestion performance. | ||
|
|
||
| On {{ecloud}}, use the [{{motlp}}](opentelemetry:/reference/motlp.md) and for self-managed use cases, you can use the [Elastic Distribution of OpenTelemetry Collector](elastic-agent:/reference/edot-collector/index.md). | ||
|
|
||
| For more details on the recommended way to set up OpenTelemetry-based data ingestion, refer to the [EDOT reference architecture](opentelemetry:/reference/architecture/index.md). | ||
|
|
||
| ## Send data to different data streams | ||
|
|
||
| By default, metrics are ingested into the `metrics-generic.otel-default` data stream. You can influence the target data stream by setting specific attributes on your data: | ||
|
|
||
| - `data_stream.dataset` or `data_stream.namespace` in attributes, with the following order of precedence: data point attribute -> scope attribute -> resource attribute | ||
| - Otherwise, if the scope name contains `/receiver/<somereceiver>`, `data_stream.dataset` is set to the receiver name. | ||
| - Otherwise, `data_stream.dataset` falls back to `generic` and `data_stream.namespace` falls back to `default`. | ||
|
|
||
| The target data stream name is constructed as `metrics-${data_stream.dataset}.otel-${data_stream.namespace}`. | ||
|
|
||
| ## Limitations | ||
|
|
||
| * Only the OTLP metrics endpoint (`/_otlp/v1/metrics`) is supported. | ||
| To ingest logs, traces, and profiles, use a distribution of the OpenTelemetry Collector that includes the [{{es}} exporter](opentelemetry:/reference/edot-collector/components/elasticsearchexporter.md), | ||
| such as the [Elastic Distribution of OpenTelemetry (EDOT) Collector](opentelemetry:/reference/edot-collector/index.md). | ||
| * Histograms are only supported in delta temporality. Set the temporality preference to delta in your SDKs, or use the [`cumulativetodelta` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/cumulativetodeltaprocessor) to avoid cumulative histograms to be dropped. | ||
felixbarny marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| * Exemplars are not supported. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One the fence about having it here as well.. will defer to @marciw
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMHO, linking from the TSDS quickstart to the OTel quickstart kinda makes sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm ok with the addition, except it kind of takes over the top of the page. But I can maybe work on making it a bit smaller -- OK as is for now