Skip to content

Expand EDOT data streams comparison to include ECS-based integration streams #469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Expand EDOT data streams comparison to include ECS-based integration streams #469

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 65 additions & 18 deletions docs/reference/compatibility/data-streams.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
navigation_title: Data streams comparison
description: Learn how EDOT optimizes telemetry storage and query performance in Elastic Observability compared to classic APM.
description: Learn how EDOT optimizes telemetry storage and query performance in Elastic Observability compared to classic APM and ECS-based integrations.
applies_to:
stack:
serverless:
Expand All @@ -12,9 +12,9 @@ products:
- id: edot-sdk
---

# OpenTelemetry data streams compared to classic APM
# OpenTelemetry data streams compared to classic APM and ECS-based integrations

The Elastic Distribution of OpenTelemetry (EDOT) stores telemetry data using a storage model optimized for OpenTelemetry signals. When `mapping::mode: otel` is enabled on the [Elasticsearch exporter](elastic-agent://reference/edot-collector/components/elasticsearchexporter.md) (which is the default setting), EDOT writes logs, traces, and metrics to specialized data streams aligned with OpenTelemetry semantics.
The Elastic Distribution of OpenTelemetry (EDOT) stores telemetry data using a storage model optimized for OpenTelemetry signals. When `mapping_mode: otel` is enabled on the Elasticsearch exporter (which is the default setting), EDOT writes logs, traces, and metrics to specialized data streams aligned with OpenTelemetry semantics.

This architecture is designed for scalable observability workloads. It supports dynamic attributes, reduces mapping complexity, and avoids issues like mapping explosions or manual dimension setup.

Expand All @@ -35,7 +35,18 @@ Metric data is stored using Elasticsearch’s [TSDS](docs-content://manage-data/
* Fast aggregations
* Automatic detection of metric dimensions (no need to manually define `time_series_dimension` in field mappings)

## Query compatibility with classic APM data streams

## Comparison with classic APM data streams

This table highlights key differences between classic Elastic APM data streams and EDOT with `mapping_mode: otel`:

| Feature | Classic APM (ECS-based) | EDOT (`mapping_mode: otel`) |
|---|---|---|
| Index mode | General-purpose data streams (logs, traces, metrics) <br><br> TSDS is not supported for classic APM. | LogsDB (logs/traces), TSDS (metrics) |
| Mapping style | Nested objects are mapped as structured fields. Some exceptions exist, such as `labels.*` and `numeric_labels.*`, where dots in field names are replaced with underscores. <br><br> ECS supports multiple field types (keyword, long, double, date, boolean, and so on) as defined in the schema. | Native OpenTelemetry fields with `passthrough`, preserving types and structure. |
| Attribute handling | Dynamic mapping. Custom attributes are stored under `labels.*` (strings) or `numeric_labels.*` (numbers); dots in field names are replaced with underscores. <br><br> See [Document examples - classic APM](#classic-apm) | Dynamic mapping with native types under `attributes.*`, preserving dots in field names. <br><br> See [Document examples - EDOT](#edot) |

### Query compatibility with classic APM data streams

EDOT is designed to make OpenTelemetry data queryable using many of the same field names as classic APM (ECS-based) data streams. This helps preserve compatibility with existing dashboards, saved searches, and queries.

Expand All @@ -44,7 +55,7 @@ Query compatibility is achieved through:
* **`passthrough` fields:** Make nested OpenTelemetry fields available at the top level so they can be queried. For example, while the service name is stored at `resource.attributes.service.name`, you can query it as `service.name` (the same field name as the one used in the classic APM data stream).
* **Field aliases:** Map fields with different names in ECS and OpenTelemetry semantic conventions to a common query name to make migration easier.

### Limitations
#### Limitations

Query compatibility is not complete:

Expand All @@ -55,20 +66,10 @@ These differences may require updates to certain queries or visualizations.

Refer to [ECS & OpenTelemetry](ecs://reference/ecs-opentelemetry.md) for details on the available aliases and field mappings.

## Comparison with classic APM data streams

This table highlights key differences between classic Elastic APM data streams and EDOT with `mapping_mode: otel`:

| Feature | Classic APM (ECS-based) | EDOT (`mapping_mode: otel`) |
|---|---|---|
| Index mode | General-purpose data streams (logs, traces, metrics) <br><br> TSDS is not supported for classic APM. | LogsDB (logs/traces), TSDS (metrics) |
| Mapping style | Nested objects are mapped as structured fields. Some exceptions exist, such as `labels.*` and `numeric_labels.*`, where dots in field names are replaced with underscores. <br><br> ECS supports multiple field types (keyword, long, double, date, boolean, etc.) as defined in the schema. | Native OpenTelemetry fields with `passthrough`, preserving types and structure. |
| Attribute handling | Dynamic mapping. Custom attributes are stored under `labels.*` (strings) or `numeric_labels.*` (numbers); dots in field names are replaced with underscores. <br><br> See [Document examples - classic APM](#classic-apm) | Dynamic mapping with native types under `attributes.*`, preserving dots in field names. <br><br> See [Document examples - EDOT](#edot) |


### Document examples

#### Classic APM:
#### Classic APM

```yaml
"@timestamp": "2025-08-14T05:29:43.922Z"
Expand All @@ -93,7 +94,7 @@ log:
level: INFO
```

#### EDOT:
#### EDOT

```yaml
"@timestamp": "2025-08-14T05:29:43.922Z"
Expand All @@ -114,4 +115,50 @@ attributes:
body:
text: "Order was successfully created"
severity_text: INFO
```
```


## Comparison with ECS-based integrations

While classic APM and EDOT represent two ingestion paths for application telemetry, Elastic’s integrations (for example Nginx, MySQL, Kubernetes) also produce ECS-based data streams for logs, metrics, and events. These use ECS mappings and integration-specific pipelines optimized for their domain.

| Stream type | Typical field layout | Custom attributes / dot notation |
|--------------|----------------------|----------------------------------|
| **Integration ECS-based** | Uses ECS mapping tailored by integration. Custom fields are added under ECS-structured objects or `.custom` objects. Dots in field names are often disallowed or normalized to underscores. | Example: `host.os.name`, `nginx.access.time` rewritten to `nginx_access_time` |
| **EDOT (OTel + passthrough)** | Stores OTel-native nested object structure (`resource.attributes`, `attributes.*`). Uses `passthrough` to expose fields at the top level for query compatibility. | Example: `attributes.cart.items: 42`, `resource.attributes.service.name: "checkout-service"` |

### Integration example (Nginx access logs)

```yaml
"@timestamp": "2025-08-14T12:00:01.123Z"
event:
dataset: nginx.access
module: nginx
host:
name: "web-1"
nginx:
access:
request: "/api/v1/items"
status_code: 200
bytes_sent: 512
user:
ip: "203.0.113.45"
```


## Summary of all data stream types

| Feature | Classic APM (ECS-based) | Integration ECS-based streams | EDOT (`mapping_mode: otel`) |
|----------|-------------------------|-------------------------------|-----------------------------|
| **Index mode** | General-purpose data streams (logs, traces, metrics); TSDS not supported | ECS-style data streams (logs, metrics, events) using integrations | LogsDB for logs/traces, TSDS for metrics |
| **Mapping style** | ECS object mappings; nested fields preserved. `labels.*` / `numeric_labels.*` flatten dots. | ECS mappings or integration-altered schemas (flattening, renaming). | OTel-native nested layout with `passthrough`, preserving types and structure. |
| **Attribute handling** | Custom values under `labels.*` / `numeric_labels.*`, dots replaced by underscores. | Integration-specific or prefixed fields. | Custom values under `attributes.*`, dots preserved. |
| **Query compatibility** | Queries target ECS field names (`service.name`, `labels.*`). | Queries assume ECS names; pipelines normalize vendor data. | `passthrough` + aliases allow ECS-style names (such as `service.name`). |
| **Compatibility limits** | N/A | Some integration fields may not align 1:1 with ECS or OTel. | Not all ECS/integration fields have aliases; label vs attribute layout differs. |


## See also

* [ECS and OpenTelemetry schema reference](ecs://reference/ecs-opentelemetry.md)
* [Logs data stream (LogsDB)](docs-content://manage-data/data-store/data-streams/logs-data-stream.md)
* [Time Series Data Stream (TSDS)](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md)
Loading