sdk/log: Deduplicate key-value collections in Record.SetBody #7002

Mojachieee · 2025-07-14T13:17:00Z

goos: darwin
goarch: arm64
pkg: go.opentelemetry.io/otel/sdk/log
cpu: Apple M2 Pro
                   │   old.txt    │               new.txt               │
                   │    sec/op    │   sec/op     vs base                │
SetBody/SetBody-12   196.5n ± 14%   365.9n ± 4%  +86.26% (p=0.000 n=10)

                   │  old.txt   │            new.txt             │
                   │    B/op    │    B/op     vs base            │
SetBody/SetBody-12   363.0 ± 0%   363.0 ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal

                   │  old.txt   │            new.txt             │
                   │ allocs/op  │ allocs/op   vs base            │
SetBody/SetBody-12   4.000 ± 0%   4.000 ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal

codecov · 2025-07-14T13:19:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.9%. Comparing base (3cd63fa) to head (e382ce9).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #7002   +/-   ##
=====================================
  Coverage   82.9%   82.9%           
=====================================
  Files        265     265           
  Lines      24682   24703   +21     
=====================================
+ Hits       20476   20499   +23     
+ Misses      3830    3828    -2     
  Partials     376     376

Files with missing lines	Coverage Δ
sdk/log/logger.go	`100.0% <100.0%> (ø)`
sdk/log/record.go	`100.0% <100.0%> (ø)`

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

MrAlias · 2025-07-14T14:50:11Z

This is not a part of the specification. Doesn't adding this move up further away from the defined specification behavior?

pellared · 2025-07-14T15:12:23Z

This is not a part of the specification. Doesn't adding this move up further away from the defined specification behavior?

@MrAlias, what do you have in mind?

"deduplication" of the key-values is part of the specification and it also involves Body of the log record.
Body is of any type (https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-body).
any can have a map<string, any> (https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#type-any).
map<string, any> does not allow duplicated keys (https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#type-mapstring-any).

Or are you concerned about the option added in #6968?

~~Or maybe #7002 (comment)?~~ EDIT: already addressed

sdk/log/record.go

CHANGELOG.md

pellared

nit comments

sdk/log/record_test.go

Co-authored-by: Robert Pająk <[email protected]>

sdk/log/record_test.go

sdk/log/record.go

pellared · 2025-07-16T07:29:06Z

With OTEP: Extending attributes to support complex values #4485 finally being merged (which was not certain at all) we most likely would not need this change as we will get rid of log.Value in favor of attribute.Value which at this point of time does not allow key-value collections. The attribute.Map and attribute.MapValue that is going to be added is planned to be handling the deduplication on creation time (see #6809).

Still, I am fine having this PR merged as I think it is better to start changing the Logs API and SDK to use attribute.Value instead of log.Value (which is going to be removed) AFTER the upcoming release (so e.g. for v1.39.0).

EDIT: I created #7034

pellared

Actually, I see that this PR is not completed.

The Body emitted via Logs API is not being deduplicated as newRecord does not call SetBody.

opentelemetry-go/sdk/log/logger.go

Line 87 in 4cb8459

body: r.Body(),

Mojachieee · 2025-07-30T12:12:42Z

Actually, I see that this PR is not completed.

The Body emitted via Logs API is not being deduplicated as newRecord does not call SetBody.

opentelemetry-go/sdk/log/logger.go

Line 87 in 4cb8459

body: r.Body(),

Good catch. This is now fixed, with a test to assert the correct behaviour

This release is the last to support [Go 1.23]. The next release will require at least [Go 1.24]. ### Added - Add native histogram exemplar support in `go.opentelemetry.io/otel/exporters/prometheus`. (#6772) - Add template attribute functions to the `go.opentelmetry.io/otel/semconv/v1.34.0` package. (#6939) - `ContainerLabel` - `DBOperationParameter` - `DBSystemParameter` - `HTTPRequestHeader` - `HTTPResponseHeader` - `K8SCronJobAnnotation` - `K8SCronJobLabel` - `K8SDaemonSetAnnotation` - `K8SDaemonSetLabel` - `K8SDeploymentAnnotation` - `K8SDeploymentLabel` - `K8SJobAnnotation` - `K8SJobLabel` - `K8SNamespaceAnnotation` - `K8SNamespaceLabel` - `K8SNodeAnnotation` - `K8SNodeLabel` - `K8SPodAnnotation` - `K8SPodLabel` - `K8SReplicaSetAnnotation` - `K8SReplicaSetLabel` - `K8SStatefulSetAnnotation` - `K8SStatefulSetLabel` - `ProcessEnvironmentVariable` - `RPCConnectRPCRequestMetadata` - `RPCConnectRPCResponseMetadata` - `RPCGRPCRequestMetadata` - `RPCGRPCResponseMetadata` - Add `ErrorType` attribute helper function to the `go.opentelmetry.io/otel/semconv/v1.34.0` package. (#6962) - Add `WithAllowKeyDuplication` in `go.opentelemetry.io/otel/sdk/log` which can be used to disable deduplication for log records. (#6968) - Add `WithCardinalityLimit` option to configure the cardinality limit in `go.opentelemetry.io/otel/sdk/metric`. (#6996, #7065, #7081, #7164, #7165, #7179) - Add `Clone` method to `Record` in `go.opentelemetry.io/otel/log` that returns a copy of the record with no shared state. (#7001) - Add experimental self-observability span and batch span processor metrics in `go.opentelemetry.io/otel/sdk/trace`. Check the `go.opentelemetry.io/otel/sdk/trace/internal/x` package documentation for more information. (#7027, #6393, #7209) - The `go.opentelemetry.io/otel/semconv/v1.36.0` package. The package contains semantic conventions from the `v1.36.0` version of the OpenTelemetry Semantic Conventions. See the [migration documentation](./semconv/v1.36.0/MIGRATION.md) for information on how to upgrade from `go.opentelemetry.io/otel/semconv/v1.34.0.`(#7032, #7041) - Add support for configuring Prometheus name translation using `WithTranslationStrategy` option in `go.opentelemetry.io/otel/exporters/prometheus`. The current default translation strategy when UTF-8 mode is enabled is `NoUTF8EscapingWithSuffixes`, but a future release will change the default strategy to `UnderscoreEscapingWithSuffixes` for compliance with the specification. (#7111) - Add experimental self-observability log metrics in `go.opentelemetry.io/otel/sdk/log`. Check the `go.opentelemetry.io/otel/sdk/log/internal/x` package documentation for more information. (#7121) - Add experimental self-observability trace exporter metrics in `go.opentelemetry.io/otel/exporters/stdout/stdouttrace`. Check the `go.opentelemetry.io/otel/exporters/stdout/stdouttrace/internal/x` package documentation for more information. (#7133) - Support testing of [Go 1.25]. (#7187) - The `go.opentelemetry.io/otel/semconv/v1.37.0` package. The package contains semantic conventions from the `v1.37.0` version of the OpenTelemetry Semantic Conventions. See the [migration documentation](./semconv/v1.37.0/MIGRATION.md) for information on how to upgrade from `go.opentelemetry.io/otel/semconv/v1.36.0.`(#7254) ### Changed - Optimize `TraceIDFromHex` and `SpanIDFromHex` in `go.opentelemetry.io/otel/sdk/trace`. (#6791) - Change `AssertEqual` in `go.opentelemetry.io/otel/log/logtest` to accept `TestingT` in order to support benchmarks and fuzz tests. (#6908) - Change `DefaultExemplarReservoirProviderSelector` in `go.opentelemetry.io/otel/sdk/metric` to use `runtime.GOMAXPROCS(0)` instead of `runtime.NumCPU()` for the `FixedSizeReservoirProvider` default size. (#7094) ### Fixed - `SetBody` method of `Record` in `go.opentelemetry.io/otel/sdk/log` now deduplicates key-value collections (`log.Value` of `log.KindMap` from `go.opentelemetry.io/otel/log`). (#7002) - Fix `go.opentelemetry.io/otel/exporters/prometheus` to not append a suffix if it's already present in metric name. (#7088) - Fix the `go.opentelemetry.io/otel/exporters/stdout/stdouttrace` self-observability component type and name. (#7195) - Fix partial export count metric in `go.opentelemetry.io/otel/exporters/stdout/stdouttrace`. (#7199) ### Deprecated - Deprecate `WithoutUnits` and `WithoutCounterSuffixes` options, preferring `WithTranslationStrategy` instead. (#7111) - Deprecate support for `OTEL_GO_X_CARDINALITY_LIMIT` environment variable in `go.opentelemetry.io/otel/sdk/metric`. Use `WithCardinalityLimit` option instead. (#7166)

Mojachieee added 5 commits July 14, 2025 13:47

add test cases for keyvalue collections with duplicates in log body

ee35e14

update SetBody to deduplicate keyvalue collections

6d291b2

refactor dedupeandlimit func into separate funcs

642daca

lint

d09263a

fix allowDuplicates bug and add test

319c086

Mojachieee requested review from MrAlias, XSAM, dashpole, pellared and dmathieu as code owners July 14, 2025 13:17

changelog

07d610f

pellared reviewed Jul 14, 2025

View reviewed changes

sdk/log/record.go Outdated Show resolved Hide resolved

pellared reviewed Jul 14, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

pellared added this to Go: Logs (GA) Jul 14, 2025

github-project-automation bot moved this to Todo in Go: Logs (GA) Jul 14, 2025

pellared assigned Mojachieee Jul 14, 2025

pellared moved this from Todo to In Progress in Go: Logs (GA) Jul 14, 2025

Mojachieee and others added 2 commits July 15, 2025 09:25

fix dedupe body using wrong function, update changelog

3e64aa9

Update CHANGELOG.md

efc658a

pellared approved these changes Jul 15, 2025

View reviewed changes

sdk/log/record_test.go Outdated Show resolved Hide resolved

sdk/log/record_test.go Outdated Show resolved Hide resolved

sdk/log/record_test.go Outdated Show resolved Hide resolved

Apply suggestions from code review

0c54eab

Co-authored-by: Robert Pająk <[email protected]>

pellared reviewed Jul 15, 2025

View reviewed changes

sdk/log/record_test.go Outdated Show resolved Hide resolved

pellared and others added 3 commits July 15, 2025 11:05

Update sdk/log/record_test.go

fbade72

Merge branch 'main' into dedup-keyvals-in-log-body

5acf33a

lint

80f0893

pellared approved these changes Jul 15, 2025

View reviewed changes

dmathieu approved these changes Jul 15, 2025

View reviewed changes

MrAlias reviewed Jul 15, 2025

View reviewed changes

sdk/log/record.go Show resolved Hide resolved

pellared requested a review from MrAlias July 16, 2025 07:29

pellared and others added 3 commits July 16, 2025 09:29

Merge branch 'main' into dedup-keyvals-in-log-body

e9bd161

Merge branch 'main' into dedup-keyvals-in-log-body

4df978d

Merge branch 'main' into dedup-keyvals-in-log-body

565eba4

Mojachieee requested a review from flc1125 as a code owner July 25, 2025 07:41

add benchmark for record.SetBody

5cddc9a

Mojachieee requested a review from pellared July 30, 2025 11:03

Merge branch 'main' into dedup-keyvals-in-log-body

4cb8459

pellared approved these changes Jul 30, 2025

View reviewed changes

pellared requested changes Jul 30, 2025

View reviewed changes

newRecord dedupes body collections & add test

6242fef

Mojachieee requested a review from pellared July 30, 2025 12:12

pellared approved these changes Jul 30, 2025

View reviewed changes

pellared added 2 commits August 12, 2025 12:57

Merge branch 'main' into dedup-keyvals-in-log-body

69af503

Merge branch 'main' into dedup-keyvals-in-log-body

12a9d7c

pellared requested a review from Copilot August 25, 2025 16:15

Merge branch 'main' into dedup-keyvals-in-log-body

e382ce9

pellared merged commit a8e1500 into open-telemetry:main Aug 25, 2025
39 checks passed

github-project-automation bot moved this from In Progress to Done in Go: Logs (GA) Aug 25, 2025

MrAlias added this to the v1.38.0 milestone Aug 26, 2025

MrAlias mentioned this pull request Aug 29, 2025

Release v1.38.0 #7271

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

sdk/log: Deduplicate key-value collections in Record.SetBody #7002

sdk/log: Deduplicate key-value collections in Record.SetBody #7002

Mojachieee commented Jul 14, 2025 •

edited

Loading

Uh oh!

codecov bot commented Jul 14, 2025 •

edited

Loading

Uh oh!

MrAlias commented Jul 14, 2025

Uh oh!

pellared commented Jul 14, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

pellared left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pellared commented Jul 16, 2025 •

edited

Loading

Uh oh!

pellared left a comment

Uh oh!

Mojachieee commented Jul 30, 2025

Uh oh!

Uh oh!

Uh oh!

sdk/log: Deduplicate key-value collections in Record.SetBody #7002

sdk/log: Deduplicate key-value collections in Record.SetBody #7002

Conversation

Mojachieee commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

MrAlias commented Jul 14, 2025

Uh oh!

pellared commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pellared left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pellared commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pellared left a comment

Choose a reason for hiding this comment

Uh oh!

Mojachieee commented Jul 30, 2025

Uh oh!

Uh oh!

Uh oh!

Mojachieee commented Jul 14, 2025 •

edited

Loading

codecov bot commented Jul 14, 2025 •

edited

Loading

pellared commented Jul 14, 2025 •

edited

Loading

pellared commented Jul 16, 2025 •

edited

Loading