SDK observability

[dashpole]

Problem Statement

Context: kubernetes/enhancements#3161 (comment)

For an application instrumented with OpenTelemetry for tracing, and using the OTLP trace exporter, it isn't currently possible to monitor (with metrics) whether or not spans are being successfully collected and exported. For example, if my SDK cannot connect to an opentelemetry collector, and isn't able to send traces, I would like to be able to measure how many traces are collected, vs how many are not sent. I would like to be able to set up SLOs to measure successful trace delivery from my applications.

Proposed Solution

After the metrics API is stable, collect metrics in the trace SDK using the metrics API. Specifics about the metrics deserve their own design, but I should be able to tell the volume of spans my application is generating, and the success rate of exporting them. This would be done via a new TracerProviderOption: WithMeterProvider(MeterProvider).

Follow semantic conventions:

• https://github.com/open-telemetry/semantic-conventions/blob/v1.36.0/docs/otel/sdk-metrics.md

Alternatives

We could add metrics to exporters individually, but most exporter-related metrics should be similar.

[MrAlias]

In the meantime, while we wait for metrics to be stable enough for this, I created this: https://github.com/MrAlias/flow

@dashpole let me know if that helps.

[dashpole]

Very cool. I'll take a look

[dashpole]

It probably isn't quite enough to meet the needs I have, but may be useful for others

[thehackercat]

we also need this.