sdk/trace: self-observability: batch span processor metrics #6393

dashpole · 2025-03-02T18:36:28Z

Adds otel.sdk.processor.span.queue.size, otel.sdk.processor.span.queue.capacity, and otel.sdk.processor.span.processed.count metrics to the trace batch span processor.

These are defined in https://github.com/open-telemetry/semantic-conventions/blob/cb11bb9bac24f4b0e95ad0f61ce01813d8ceada8/docs/otel/sdk-metrics.md, and are experimental. Because of this, metrics are behind the OTEL_GO_X_SELF_OBSERVABILITY feature gate.

Given the feature is experimental, it always uses the global meterprovider when enabled.

codecov · 2025-03-02T18:38:29Z

Codecov Report

❌ Patch coverage is 80.64516% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.0%. Comparing base (c5e68b2) to head (e162018).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
sdk/trace/batch_span_processor.go	80.3%	8 Missing and 4 partials ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #6393   +/-   ##
=====================================
  Coverage   82.9%   83.0%           
=====================================
  Files        264     264           
  Lines      24574   24634   +60     
=====================================
+ Hits       20392   20451   +59     
+ Misses      3802    3800    -2     
- Partials     380     383    +3

Files with missing lines	Coverage Δ
sdk/trace/provider.go	`86.4% <ø> (+1.0%)`	⬆️
sdk/trace/tracer.go	`95.9% <100.0%> (ø)`
sdk/trace/batch_span_processor.go	`80.4% <80.3%> (+2.1%)`	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

I would like to be able to use a private option in #6393 in tests, and decided to split this refactoring out into its own PR. This moves the batch span processor benchmarks into benchmark_test.go, and replaces one instance of the tracetest.NewInMemoryExporter with a different test exporter implementation. It then moves most unit tests from `trace_test` to the main `trace` package.

dashpole · 2025-07-16T14:14:34Z

I remember why I didn't get this merged now... Unit tests for the queue size are hard to write without race conditions.

sdk/trace/batch_span_processor.go

sdk/trace/batch_span_processor_test.go

pellared

Implementation LGTM. I think it would be nice to refactor the tests. Maybe I am too pedantic...

Co-authored-by: Robert Pająk <[email protected]>

sdk/trace/batch_span_processor_test.go

Co-authored-by: Robert Pająk <[email protected]>

This release is the last to support [Go 1.23]. The next release will require at least [Go 1.24]. ### Added - Add native histogram exemplar support in `go.opentelemetry.io/otel/exporters/prometheus`. (#6772) - Add template attribute functions to the `go.opentelmetry.io/otel/semconv/v1.34.0` package. (#6939) - `ContainerLabel` - `DBOperationParameter` - `DBSystemParameter` - `HTTPRequestHeader` - `HTTPResponseHeader` - `K8SCronJobAnnotation` - `K8SCronJobLabel` - `K8SDaemonSetAnnotation` - `K8SDaemonSetLabel` - `K8SDeploymentAnnotation` - `K8SDeploymentLabel` - `K8SJobAnnotation` - `K8SJobLabel` - `K8SNamespaceAnnotation` - `K8SNamespaceLabel` - `K8SNodeAnnotation` - `K8SNodeLabel` - `K8SPodAnnotation` - `K8SPodLabel` - `K8SReplicaSetAnnotation` - `K8SReplicaSetLabel` - `K8SStatefulSetAnnotation` - `K8SStatefulSetLabel` - `ProcessEnvironmentVariable` - `RPCConnectRPCRequestMetadata` - `RPCConnectRPCResponseMetadata` - `RPCGRPCRequestMetadata` - `RPCGRPCResponseMetadata` - Add `ErrorType` attribute helper function to the `go.opentelmetry.io/otel/semconv/v1.34.0` package. (#6962) - Add `WithAllowKeyDuplication` in `go.opentelemetry.io/otel/sdk/log` which can be used to disable deduplication for log records. (#6968) - Add `WithCardinalityLimit` option to configure the cardinality limit in `go.opentelemetry.io/otel/sdk/metric`. (#6996, #7065, #7081, #7164, #7165, #7179) - Add `Clone` method to `Record` in `go.opentelemetry.io/otel/log` that returns a copy of the record with no shared state. (#7001) - Add experimental self-observability span and batch span processor metrics in `go.opentelemetry.io/otel/sdk/trace`. Check the `go.opentelemetry.io/otel/sdk/trace/internal/x` package documentation for more information. (#7027, #6393, #7209) - The `go.opentelemetry.io/otel/semconv/v1.36.0` package. The package contains semantic conventions from the `v1.36.0` version of the OpenTelemetry Semantic Conventions. See the [migration documentation](./semconv/v1.36.0/MIGRATION.md) for information on how to upgrade from `go.opentelemetry.io/otel/semconv/v1.34.0.`(#7032, #7041) - Add support for configuring Prometheus name translation using `WithTranslationStrategy` option in `go.opentelemetry.io/otel/exporters/prometheus`. The current default translation strategy when UTF-8 mode is enabled is `NoUTF8EscapingWithSuffixes`, but a future release will change the default strategy to `UnderscoreEscapingWithSuffixes` for compliance with the specification. (#7111) - Add experimental self-observability log metrics in `go.opentelemetry.io/otel/sdk/log`. Check the `go.opentelemetry.io/otel/sdk/log/internal/x` package documentation for more information. (#7121) - Add experimental self-observability trace exporter metrics in `go.opentelemetry.io/otel/exporters/stdout/stdouttrace`. Check the `go.opentelemetry.io/otel/exporters/stdout/stdouttrace/internal/x` package documentation for more information. (#7133) - Support testing of [Go 1.25]. (#7187) - The `go.opentelemetry.io/otel/semconv/v1.37.0` package. The package contains semantic conventions from the `v1.37.0` version of the OpenTelemetry Semantic Conventions. See the [migration documentation](./semconv/v1.37.0/MIGRATION.md) for information on how to upgrade from `go.opentelemetry.io/otel/semconv/v1.36.0.`(#7254) ### Changed - Optimize `TraceIDFromHex` and `SpanIDFromHex` in `go.opentelemetry.io/otel/sdk/trace`. (#6791) - Change `AssertEqual` in `go.opentelemetry.io/otel/log/logtest` to accept `TestingT` in order to support benchmarks and fuzz tests. (#6908) - Change `DefaultExemplarReservoirProviderSelector` in `go.opentelemetry.io/otel/sdk/metric` to use `runtime.GOMAXPROCS(0)` instead of `runtime.NumCPU()` for the `FixedSizeReservoirProvider` default size. (#7094) ### Fixed - `SetBody` method of `Record` in `go.opentelemetry.io/otel/sdk/log` now deduplicates key-value collections (`log.Value` of `log.KindMap` from `go.opentelemetry.io/otel/log`). (#7002) - Fix `go.opentelemetry.io/otel/exporters/prometheus` to not append a suffix if it's already present in metric name. (#7088) - Fix the `go.opentelemetry.io/otel/exporters/stdout/stdouttrace` self-observability component type and name. (#7195) - Fix partial export count metric in `go.opentelemetry.io/otel/exporters/stdout/stdouttrace`. (#7199) ### Deprecated - Deprecate `WithoutUnits` and `WithoutCounterSuffixes` options, preferring `WithTranslationStrategy` instead. (#7111) - Deprecate support for `OTEL_GO_X_CARDINALITY_LIMIT` environment variable in `go.opentelemetry.io/otel/sdk/metric`. Use `WithCardinalityLimit` option instead. (#7166)

dashpole mentioned this pull request Mar 4, 2025

Move trace sdk tests from trace_test into trace package #6400

Merged

dashpole force-pushed the trace_sdk_processor_metrics branch from 68bcd1b to 1809feb Compare June 24, 2025 01:00

MrAlias mentioned this pull request Jul 10, 2025

SDK observability #2547

Open

dashpole force-pushed the trace_sdk_processor_metrics branch 2 times, most recently from 17fe74e to 40e25fb Compare July 15, 2025 15:45

dashpole mentioned this pull request Jul 15, 2025

otelconv only generates synchronous instruments #7029

Closed

pellared mentioned this pull request Jul 16, 2025

sdk/trace: self-observability: span metrics #7027

Merged

dashpole force-pushed the trace_sdk_processor_metrics branch 2 times, most recently from 1ccad8b to 10f11a9 Compare July 16, 2025 19:29

dashpole force-pushed the trace_sdk_processor_metrics branch from 4544d8d to 885da03 Compare July 30, 2025 14:35

pellared reviewed Jul 30, 2025

View reviewed changes

sdk/trace/batch_span_processor.go Outdated Show resolved Hide resolved

dashpole force-pushed the trace_sdk_processor_metrics branch from 7134f27 to 212f479 Compare July 30, 2025 16:53

pellared reviewed Jul 30, 2025

View reviewed changes

sdk/trace/batch_span_processor.go Outdated Show resolved Hide resolved

dashpole force-pushed the trace_sdk_processor_metrics branch from 212f479 to fceaf98 Compare July 30, 2025 16:59

pellared reviewed Aug 4, 2025

View reviewed changes

sdk/trace/batch_span_processor.go Outdated Show resolved Hide resolved

dashpole added 3 commits August 6, 2025 16:56

add experimental batch span processor metrics

a0d68e7

address comments

d20963d

remove noError

e0d4b48

dashpole force-pushed the trace_sdk_processor_metrics branch from 88ba37f to e0d4b48 Compare August 6, 2025 16:56

dashpole commented Aug 6, 2025

View reviewed changes

sdk/trace/batch_span_processor.go Outdated Show resolved Hide resolved

dashpole added 2 commits August 6, 2025 13:03

Fix lint

901fda8

Merge branch 'main' into trace_sdk_processor_metrics

ab461a9

pellared reviewed Aug 7, 2025

View reviewed changes

sdk/trace/batch_span_processor_test.go Outdated Show resolved Hide resolved

sdk/trace/batch_span_processor_test.go Outdated Show resolved Hide resolved

sdk/trace/batch_span_processor_test.go Outdated Show resolved Hide resolved

sdk/trace/batch_span_processor_test.go Show resolved Hide resolved

pellared reviewed Aug 7, 2025

View reviewed changes

dashpole and others added 3 commits August 7, 2025 09:48

Update sdk/trace/batch_span_processor_test.go

8369e15

Co-authored-by: Robert Pająk <[email protected]>

assert against entire ScopeMetrics

f1c41eb

waitForSpans respects cancellation

bd6e7b2

pellared approved these changes Aug 11, 2025

View reviewed changes

sdk/trace/batch_span_processor_test.go Outdated Show resolved Hide resolved

pellared changed the title ~~Trace sdk processor metrics~~ sdk/trace: self-observability: batch span processor metrics Aug 11, 2025

dashpole and others added 5 commits August 11, 2025 09:54

Update sdk/trace/batch_span_processor_test.go

33868c2

Co-authored-by: Robert Pająk <[email protected]>

Merge branch 'main' into trace_sdk_processor_metrics

f57d513

Merge branch 'main' into trace_sdk_processor_metrics

cbc1512

Merge branch 'main' into trace_sdk_processor_metrics

23716b7

Merge branch 'main' into trace_sdk_processor_metrics

e162018

dashpole merged commit fcc3417 into open-telemetry:main Aug 11, 2025
32 of 33 checks passed

dashpole deleted the trace_sdk_processor_metrics branch August 11, 2025 20:15

MrAlias added this to the v1.38.0 milestone Aug 20, 2025

MrAlias mentioned this pull request Aug 29, 2025

Release v1.38.0 #7271

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

sdk/trace: self-observability: batch span processor metrics #6393

sdk/trace: self-observability: batch span processor metrics #6393

dashpole commented Mar 2, 2025 •

edited

Loading

Uh oh!

codecov bot commented Mar 2, 2025 •

edited

Loading

Uh oh!

dashpole commented Jul 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pellared left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sdk/trace: self-observability: batch span processor metrics #6393

sdk/trace: self-observability: batch span processor metrics #6393

Conversation

dashpole commented Mar 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dashpole commented Jul 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pellared left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dashpole commented Mar 2, 2025 •

edited

Loading

codecov bot commented Mar 2, 2025 •

edited

Loading

pellared left a comment •

edited

Loading