Support adding per-request custom attributes to HTTP server metrics

[tammy-baylis-swi]

What problem do you want to solve?

It would be great if existing HTTP server instrumentors that write metrics (Flask, Django, Falcon, WSGI, ASGI) supported adding per-request custom attributes to those metrics.

It would help these use cases, some based on #1801, #3405, and open-telemetry/opentelemetry-go-contrib#195:

1. Querying metrics by a low-cardinality entity ID that's only available after request received and auth validated
2. Querying by customizable values that depend on current request, without changing span name
3. Grouping targets that use the same API name or version, but not necessarily the same service
4. Using existing metrics without having to calculate and record additional stats
5. Enhancing metrics at instrumentor origin early, instead of downstream by otel-collector

In the semconv, there seems to be nothing that explicitly says “never add attributes that aren’t in semconv”. I do want to note these parts:

1. This part says that otel.* attributes must be semconv-approved.
2. This says “If an appropriate name does not exists you will need to come up with a new name” regarding attributes, metrics, events, etc by application developers. Just make sure to namespace as much as possible and be mindful of internal vs company usage.
3. Though for db span attributes, it mentions “When additional attributes are added” implying it’s generally ok, again as long as we namespace.

Describe the solution you'd like

One approach could be to implement a new Labeler utility like the Go net/http instrumentation. It would need to be thread-safe and exist for the request in the current context (e.g. contextvars). Server code that's instrumented or custom distros would use the current Labeler or a new one to add KVs. The HTTP server instrumentors would check for any Labeler in context, read its KVs, and merge them into the attributes currently used to write HTTP metrics.

Describe alternatives you've considered

I've thought about using OTel Python's Baggage for storing custom attributes, since the API already exists and is based on W3C and OTel standards. However, that would come with challenges:

1. set_baggage can be called by user or custom distro code. get_baggage can be called by an HTTP instrumentor to merge attributes. But remove_baggage should be called by someone prevent HTTP propagation (of potentially sensitive info in some cases), which is the primary purpose of W3C baggage. Maybe it’s the instrumentors that should remove, but it's a bit odd if they’re not the ones doing set_baggage.
2. set / get / remove need to be done with a reserved key that needs to be well-documented. otel-metrics-custom-attributes?
3. Python’s Baggage API docs need to be updated; see this issue. Setting baggage must be done in two lines by distro or user code: token = context.attach(baggage.set_baggage("foo", "bar")) then context.detach(token).

Additional Context

Inspired by Go net/http instrumentor Labeler introduced with open-telemetry/opentelemetry-go-contrib#195

Relates to

• [asgi] Allow to attach custom attributes #1801
• [Flask] request_hook for metrics attributes #3405

Would you like to implement a fix?

Yes

Tasks

• Add Labeler implementation #3702
• Update Flask instrumentation to merge attributes set by Labeler #3703
• Update Django instrumentation to merge attributes set by Labeler #3704
• Update Falcon instrumentation to merge attributes set by Labeler #3705
• Update WSGI instrumentation to merge attributes set by Labeler #3706

[tammy-baylis-swi]

cc'ing @aabmass following today's python sig meeting