Skip to content

Commit f7b7a6f

Browse files
isaacaflores2isaacaflores2
authored
[exporter/elasticsearch] Update OTEL SemConv version and Fix ECS mapping for messaging.destination.name (open-telemetry#43884)
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description - Fixes the ECS mapping for the OTEL sem conv `messaging.destination.name` field which should map to `span.message.queue.name` or `transaction.message.queue.name` based on the trace type (Elastic span or transaction) * This mapping was added in a recent PR open-telemetry#43807 - Add the current latest OTEL SemConv version `v1.37.0` to introduce `deployment.environment.name` which maps to `service.environment`. - The prior version of SemConv is kept to maintain backwards compatibility for deprecated attributes. <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#43805 and open-telemetry#43806 <!--Describe what testing was performed and which tests were added.--> #### Testing - Updated unit tests - Setup a local collector with the elasticsearch exporter to validate indexed documents <!--Describe the documentation added.--> #### Documentation - Updates the `README.md` with OTEL semconv attributes that were recently added to the ECS mappings
1 parent 9cf8ee6 commit f7b7a6f

File tree

4 files changed

+285
-141
lines changed

4 files changed

+285
-141
lines changed

‎.chloggen/es-exporter-update-semconv-version-ecs-mappping.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Use this changelog template to create an entry for release notes.
2+
3+
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
4+
change_type: enhancement
5+
6+
# The name of the component, or a single word describing the area of concern, (e.g. receiver/filelog)
7+
component: exporter/elasticsearch
8+
9+
# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
10+
note: Add support for latest OTEL SemConv version and fix Elasticsearch exporter ECS mapping for message.destination.name which is different for Elastic spans or transactions
11+
12+
# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
13+
issues: [43805,43806]
14+
15+
# (Optional) One or more lines of additional information to render under the primary note.
16+
# These lines will be padded with 2 spaces and then inserted directly into the document.
17+
# Use pipe (|) for multiline entries.
18+
subtext:
19+
20+
# If your change doesn't affect end users or the exported elements of any package,
21+
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
22+
# Optional: The change log or logs in which this entry should be included.
23+
# e.g. '[user]' or '[user, api]'
24+
# Include 'user' if the change is relevant to end users.
25+
# Include 'api' if there is a change to a library API.
26+
# Default: '[user]'
27+
change_logs: [user]

‎exporter/elasticsearchexporter/README.md‎

Lines changed: 74 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -418,44 +418,80 @@ exporters:
418418

419419
`elasticsearchexporter` follows ECS mapping defined here: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model-appendix.md#elastic-common-schema
420420

421-
When `mode` is set to `ecs`, `elasticsearchexporter` performs conversions for resource-level attributes from their Semantic Conventions (SemConv) names to equivalent Elastic Common Schema (ECS) names.
421+
When `mode` is set to `ecs`, `elasticsearchexporter` performs conversions for resource-level and record-level (log or trace) attributes from their Semantic Conventions (SemConv) names to equivalent Elastic Common Schema (ECS) names.
422422

423423
If the target ECS field name is specified as an empty string (`""`), the converter will neither convert the SemConv key to the equivalent ECS name nor pass through the SemConv key as-is to become the ECS name.
424424

425425
When "Preserved" is true, the attribute will be preserved in the payload and duplicated as mapped to its ECS equivalent.
426426

427-
| Semantic Convention Name | ECS Name | Preserve |
428-
|--------------------------|-----------------------------|----------|
429-
| cloud.platform | cloud.service.name | false |
430-
| container.image.tags | container.image.tag | false |
431-
| deployment.environment | service.environment | false |
432-
| host.arch | host.architecture | false |
433-
| host.name | host.hostname | true |
434-
| k8s.cluster.name | orchestrator.cluster.name | false |
435-
| k8s.container.name | kubernetes.container.name | false |
436-
| k8s.cronjob.name | kubernetes.cronjob.name | false |
437-
| k8s.daemonset.name | kubernetes.daemonset.name | false |
438-
| k8s.deployment.name | kubernetes.deployment.name | false |
439-
| k8s.job.name | kubernetes.job.name | false |
440-
| k8s.namespace.name | kubernetes.namespace | false |
441-
| k8s.node.name | kubernetes.node.name | false |
442-
| k8s.pod.name | kubernetes.pod.name | false |
443-
| k8s.pod.uid | kubernetes.pod.uid | false |
444-
| k8s.replicaset.name | kubernetes.replicaset.name | false |
445-
| k8s.statefulset.name | kubernetes.statefulset.name | false |
446-
| os.description | host.os.full | false |
447-
| os.name | host.os.name | false |
448-
| os.type | host.os.platform | false |
449-
| os.version | host.os.version | false |
450-
| process.executable.path | process.executable | false |
451-
| process.runtime.name | service.runtime.name | false |
452-
| process.runtime.version | service.runtime.version | false |
453-
| service.instance.id | service.node.name | false |
454-
| telemetry.distro.name | "" | false |
455-
| telemetry.distro.version | "" | false |
456-
| telemetry.sdk.language | "" | false |
457-
| telemetry.sdk.name | "" | false |
458-
| telemetry.sdk.version | "" | false |
427+
When more than one SemConv attribute maps to the same ECS attribute, the converter will map all attributes to the same ECS name.
428+
This is mean to support backwards compatibility for SemConv attributes that have been renamed/deprecated.
429+
The value of the last-mapped attribute will take precedence.
430+
431+
### Resource attribute mapping
432+
433+
| Semantic Convention Name | ECS Name | Preserve |
434+
|-----------------------------|-----------------------------|----------|
435+
| client.address | client.ip | false |
436+
| cloud.platform | cloud.service.name | false |
437+
| container.image.tags | container.image.tag | false |
438+
| deployment.environment | service.environment | false |
439+
| deployment.environment.name | service.environment | false |
440+
| faas.instance | faas.id | false |
441+
| faas.trigger | faas.trigger.type | false |
442+
| host.arch | host.architecture | false |
443+
| host.name | host.hostname | true |
444+
| k8s.cluster.name | orchestrator.cluster.name | false |
445+
| k8s.container.name | kubernetes.container.name | false |
446+
| k8s.cronjob.name | kubernetes.cronjob.name | false |
447+
| k8s.daemonset.name | kubernetes.daemonset.name | false |
448+
| k8s.deployment.name | kubernetes.deployment.name | false |
449+
| k8s.job.name | kubernetes.job.name | false |
450+
| k8s.namespace.name | kubernetes.namespace | false |
451+
| k8s.node.name | kubernetes.node.name | false |
452+
| k8s.pod.name | kubernetes.pod.name | false |
453+
| k8s.pod.uid | kubernetes.pod.uid | false |
454+
| k8s.replicaset.name | kubernetes.replicaset.name | false |
455+
| k8s.statefulset.name | kubernetes.statefulset.name | false |
456+
| os.description | host.os.full | false |
457+
| os.name | host.os.name | false |
458+
| os.type | host.os.platform | false |
459+
| os.version | host.os.version | false |
460+
| process.command_line | process.args | false |
461+
| process.executable.name | process.title | false |
462+
| process.executable.path | process.executable | false |
463+
| process.parent.pid | process.parent.pid | false |
464+
| process.runtime.name | service.runtime.name | false |
465+
| process.runtime.version | service.runtime.version | false |
466+
| service.instance.id | service.node.name | false |
467+
| source.address | source.ip | false |
468+
| telemetry.distro.name | "" | false |
469+
| telemetry.distro.version | "" | false |
470+
| telemetry.sdk.language | "" | false |
471+
| telemetry.sdk.name | "" | false |
472+
| telemetry.sdk.version | "" | false |
473+
474+
### Log record attribute mapping
475+
476+
| Semantic Convention Name | ECS Name | Preserve |
477+
|--------------------------|---------------------------------|----------|
478+
| event.name | event.action | false |
479+
| exception.message | error.message | false |
480+
| exception.stacktrace | error.stacktrace | false |
481+
| exception.type | error.type | false |
482+
| exception.escaped | event.error.exception.handled | false |
483+
| http.response.body.size | http.response.encoded_body_size | false |
484+
485+
### Span attribute mapping
486+
487+
| Semantic Convention Name | ECS Name | Preserve |
488+
|----------------------------|-----------------------------------------------------------|----------|
489+
| messaging.operation.name | span.action | false |
490+
| db.system | span.db.type | false |
491+
| db.namespace | span.db.instance | false |
492+
| db.query.text | span.db.statement | false |
493+
| http.response.body.size | http.response.encoded_body_size | false |
494+
459495

460496
### Compound Mapping
461497

@@ -508,6 +544,11 @@ Otherwise, it is mapped to an empty string ("").
508544

509545
In case the record contains `timestamp`, this value is used. Otherwise, the `observed timestamp` is used.
510546

547+
### `messaging.destination.name`
548+
549+
Maps to `span.message.queue.name` for regular spans, but to `transaction.message.queue.name` when the `processor.event` attribute equals "transaction".
550+
This attribute is only applicable at the trace level.
551+
511552
## Setting a document id dynamically
512553

513554
The `logs_dynamic_id` setting allows users to set the document ID dynamically based on a log record attribute.

‎exporter/elasticsearchexporter/model.go‎

Lines changed: 54 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,8 @@ import (
1414
"go.opentelemetry.io/collector/pdata/plog"
1515
"go.opentelemetry.io/collector/pdata/pprofile"
1616
"go.opentelemetry.io/collector/pdata/ptrace"
17-
semconv "go.opentelemetry.io/otel/semconv/v1.22.0"
17+
semconv22 "go.opentelemetry.io/otel/semconv/v1.22.0"
18+
semconv "go.opentelemetry.io/otel/semconv/v1.37.0"
1819

1920
"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter/internal/datapoints"
2021
"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter/internal/elasticsearch"
@@ -31,43 +32,44 @@ import (
3132
// neither convert the SemConv key to the equivalent ECS name nor pass-through the
3233
// SemConv key as-is to become the ECS name.
3334
var resourceAttrsConversionMap = map[string]string{
34-
string(semconv.ServiceInstanceIDKey): "service.node.name",
35-
string(semconv.DeploymentEnvironmentKey): "service.environment",
36-
string(semconv.TelemetrySDKNameKey): "",
37-
string(semconv.TelemetrySDKLanguageKey): "",
38-
string(semconv.TelemetrySDKVersionKey): "",
39-
string(semconv.TelemetryDistroNameKey): "",
40-
string(semconv.TelemetryDistroVersionKey): "",
41-
string(semconv.CloudPlatformKey): "cloud.service.name",
42-
string(semconv.ContainerImageTagsKey): "container.image.tag",
43-
string(semconv.HostNameKey): "host.hostname",
44-
string(semconv.HostArchKey): "host.architecture",
45-
string(semconv.ProcessParentPIDKey): "process.parent.pid",
46-
string(semconv.ProcessExecutableNameKey): "process.title",
47-
string(semconv.ProcessExecutablePathKey): "process.executable",
48-
string(semconv.ProcessCommandLineKey): "process.args",
49-
string(semconv.ProcessRuntimeNameKey): "service.runtime.name",
50-
string(semconv.ProcessRuntimeVersionKey): "service.runtime.version",
51-
string(semconv.OSNameKey): "host.os.name",
52-
string(semconv.OSTypeKey): "host.os.platform",
53-
string(semconv.OSDescriptionKey): "host.os.full",
54-
string(semconv.OSVersionKey): "host.os.version",
55-
string(semconv.ClientAddressKey): "client.ip",
56-
string(semconv.SourceAddressKey): "source.ip",
57-
string(semconv.K8SDeploymentNameKey): "kubernetes.deployment.name",
58-
string(semconv.K8SNamespaceNameKey): "kubernetes.namespace",
59-
string(semconv.K8SNodeNameKey): "kubernetes.node.name",
60-
string(semconv.K8SPodNameKey): "kubernetes.pod.name",
61-
string(semconv.K8SPodUIDKey): "kubernetes.pod.uid",
62-
string(semconv.K8SJobNameKey): "kubernetes.job.name",
63-
string(semconv.K8SCronJobNameKey): "kubernetes.cronjob.name",
64-
string(semconv.K8SStatefulSetNameKey): "kubernetes.statefulset.name",
65-
string(semconv.K8SReplicaSetNameKey): "kubernetes.replicaset.name",
66-
string(semconv.K8SDaemonSetNameKey): "kubernetes.daemonset.name",
67-
string(semconv.K8SContainerNameKey): "kubernetes.container.name",
68-
string(semconv.K8SClusterNameKey): "orchestrator.cluster.name",
69-
string(semconv.FaaSInstanceKey): "faas.id",
70-
string(semconv.FaaSTriggerKey): "faas.trigger.type",
35+
string(semconv.ServiceInstanceIDKey): "service.node.name",
36+
string(semconv22.DeploymentEnvironmentKey): "service.environment",
37+
string(semconv.DeploymentEnvironmentNameKey): "service.environment",
38+
string(semconv.TelemetrySDKNameKey): "",
39+
string(semconv.TelemetrySDKLanguageKey): "",
40+
string(semconv.TelemetrySDKVersionKey): "",
41+
string(semconv.TelemetryDistroNameKey): "",
42+
string(semconv.TelemetryDistroVersionKey): "",
43+
string(semconv.CloudPlatformKey): "cloud.service.name",
44+
string(semconv.ContainerImageTagsKey): "container.image.tag",
45+
string(semconv.HostNameKey): "host.hostname",
46+
string(semconv.HostArchKey): "host.architecture",
47+
string(semconv.ProcessParentPIDKey): "process.parent.pid",
48+
string(semconv.ProcessExecutableNameKey): "process.title",
49+
string(semconv.ProcessExecutablePathKey): "process.executable",
50+
string(semconv.ProcessCommandLineKey): "process.args",
51+
string(semconv.ProcessRuntimeNameKey): "service.runtime.name",
52+
string(semconv.ProcessRuntimeVersionKey): "service.runtime.version",
53+
string(semconv.OSNameKey): "host.os.name",
54+
string(semconv.OSTypeKey): "host.os.platform",
55+
string(semconv.OSDescriptionKey): "host.os.full",
56+
string(semconv.OSVersionKey): "host.os.version",
57+
string(semconv.ClientAddressKey): "client.ip",
58+
string(semconv.SourceAddressKey): "source.ip",
59+
string(semconv.K8SDeploymentNameKey): "kubernetes.deployment.name",
60+
string(semconv.K8SNamespaceNameKey): "kubernetes.namespace",
61+
string(semconv.K8SNodeNameKey): "kubernetes.node.name",
62+
string(semconv.K8SPodNameKey): "kubernetes.pod.name",
63+
string(semconv.K8SPodUIDKey): "kubernetes.pod.uid",
64+
string(semconv.K8SJobNameKey): "kubernetes.job.name",
65+
string(semconv.K8SCronJobNameKey): "kubernetes.cronjob.name",
66+
string(semconv.K8SStatefulSetNameKey): "kubernetes.statefulset.name",
67+
string(semconv.K8SReplicaSetNameKey): "kubernetes.replicaset.name",
68+
string(semconv.K8SDaemonSetNameKey): "kubernetes.daemonset.name",
69+
string(semconv.K8SContainerNameKey): "kubernetes.container.name",
70+
string(semconv.K8SClusterNameKey): "orchestrator.cluster.name",
71+
string(semconv.FaaSInstanceKey): "faas.id",
72+
string(semconv.FaaSTriggerKey): "faas.trigger.type",
7173
}
7274

7375
// resourceAttrsToPreserve contains conventions that should be preserved in ECS mode.
@@ -210,7 +212,7 @@ func (ecsModeEncoder) encodeLog(
210212
string(semconv.ExceptionMessageKey): "error.message",
211213
string(semconv.ExceptionStacktraceKey): "error.stacktrace",
212214
string(semconv.ExceptionTypeKey): "error.type",
213-
string(semconv.ExceptionEscapedKey): "event.error.exception.handled",
215+
string(semconv22.ExceptionEscapedKey): "event.error.exception.handled",
214216
string(semconv.HTTPResponseBodySizeKey): "http.response.encoded_body_size",
215217
}
216218
encodeAttributesECSMode(&document, record.Attributes(), recordAttrsConversionMap, resourceAttrsToPreserve)
@@ -254,12 +256,20 @@ func (ecsModeEncoder) encodeSpan(
254256
encodeAttributesECSMode(&document, ec.scope.Attributes(), scopeAttrsConversionMap, resourceAttrsToPreserve)
255257

256258
// Finally, try to map record-level attributes to ECS fields.
259+
260+
// determine the correct message queue name based on the trace type (Elastic span or transaction)
261+
messageQueueName := "span.message.queue.name"
262+
processor, _ := span.Attributes().Get("processor.event")
263+
if processor.Str() == "transaction" {
264+
messageQueueName = "transaction.message.queue.name"
265+
}
266+
257267
spanAttrsConversionMap := map[string]string{
258-
string(semconv.MessagingDestinationNameKey): "span.message.queue.name",
259-
"messaging.operation.name": "span.action",
260-
string(semconv.DBSystemKey): "span.db.type",
261-
"db.namespace": "span.db.instance",
262-
"db.query.text": "span.db.statement",
268+
string(semconv.MessagingDestinationNameKey): messageQueueName,
269+
string(semconv.MessagingOperationNameKey): "span.action",
270+
string(semconv22.DBSystemKey): "span.db.type",
271+
string(semconv.DBNamespaceKey): "span.db.instance",
272+
string(semconv.DBQueryTextKey): "span.db.statement",
263273
string(semconv.HTTPResponseBodySizeKey): "http.response.encoded_body_size",
264274
}
265275

0 commit comments

Comments
 (0)