Align Server Observation with OTel semantic convention

[vmeunier]

Hello,

There seems to be a divergence between https://opentelemetry.io/docs/specs/semconv/http/http-spans/#http-server-semantic-conventions and what Spring offers by default in ServerHttpObservationDocumentation.

For example :

• method should be http.request.method
• status should be http.response.status_code and should be an int and not a String
• uri should be http.route

I haven't listed all the differences, but these are the main ones I happened to notice.

I'd be happy to contribute if necessary.

[bclozel]

That is a breaking change so we could only consider this for 7.0. Also, this will cause upgrade concerns as 6.x apps and 7.x apps will not ship the same metadata and this is likely to break existing dashboards.

@shakuzen do you think we should reconsider our existing metadata for 7.0 (at least the stable conventions) or should this be handled by https://github.com/micrometer-metrics/micrometer-otel-conventions-experimental ?

[vmeunier]

Can we consider duplicating the keys to avoid making breaking changes, and mark the old ones as deprecated ?

[bclozel]

@vmeunier is there a way to mark attributes as deprecated and show that status in dashboards? That would be interesting.
Deprecating that in our own conventions would not tell much to most developers as they won't see it unless they extend those classes.

[vmeunier]

Not that I know of unfortunately, since attributes are only KeyValue pairs... And you're right, adding that info in your own code, no one will see it, so it's probably not the best idea.

Note : I just checked the ClientHttpObservationDocumentation class, and there are several keys that are not named correctly there as well.

[shakuzen]

That is a breaking change so we could only consider this for 7.0. Also, this will cause upgrade concerns as 6.x apps and 7.x apps will not ship the same metadata and this is likely to break existing dashboards.

@shakuzen do you think we should reconsider our existing metadata for 7.0 (at least the stable conventions) or should this be handled by https://github.com/micrometer-metrics/micrometer-otel-conventions-experimental ?

It's a tough call for the reasons you mentioned. It's worth pointing out for historical context that at the time the mentioned instrumentation was made GA in Spring Framework, there were no stable OTel semantic conventions and if you aligned with what was there then, it would not match what is stable now. Hence 7.0 is the first opportunity Framework has to align its default convention with the stable HTTP semantic conventions defined by OTel. In addition, you would need to consider the "requirement level" mentioned in the OTel semantic conventions and decide which your instrumentation should contain or not. The span convention was linked, but there's also the metrics convention for HTTP servers to consider.

We have some on-going work to explore how we can ease configuring an ObservationConvention that matches some version of OTel semantic conventions - see micrometer-metrics/micrometer#5794. What concrete form that will take is not decided yet. https://github.com/micrometer-metrics/micrometer-otel-conventions-experimental was an experiment we worked on related to this, but we may end up taking a different approach. If Framework were to change its default convention in a breaking way in 7.0, it would be nice for users if there were an easy way to switch back to the 6.x convention, so the decision about upgrading one app's Framework version and the observability convention it uses can be made separately.

It's also worth noting that users can already configure any convention they would like, so they can make decisions such as what version of e.g. OTel semantic conventions to adopt and what "requirement level" they want. That is documented in this section of the Framework docs.

Can we consider duplicating the keys to avoid making breaking changes

That can be pragmatic, but it's best as a temporary measure until hopefully all applications in an organization can be transitioned to the new convention and consumers of observability data (dashboards, alerts, etc.) can be migrated. More observability data isn't free (CPU cycles, memory, network, storage), and making the default convention produce twice as much data can result in a substantial cost increase for users, while most users most of the time will probably want either one convention or another, rather than pay to have both all the time.

In summary, maybe it makes sense for Framework to wait a bit to see how micrometer-metrics/micrometer#5794 ends up. Otherwise, providing multiple convention implementations and an easy way to switch between them may be pragmatic.

[jonatan-ivanov]

I would like to mention this from the first line in the docs:

Status: Stable, Unless otherwise specified.

Unfortunately, the stable/development markers are images in the html so it's not easy to search (you can search in the markdown) 🙂 but there are lots of attributes that has the development marker. I guess this really means that the http semconv as a whole is not stable yet, the question is: is everything stable that we need?

I guess probably not since:

• Span status is not stable
• IIRC the network semconv that HTTP depends on has an unstable attribute

[bclozel]

Thanks a lot for you feedback @shakuzen @jonatan-ivanov - this is not an easy situation to resolve. We are somehow stuck between an existing user base with dashboards/tooling in production and inflicting breaking changes in waves to the community without a clear deadline.

I'll close this issue in favour of micrometer-metrics/micrometer#5794 for now. I'm willing to work on OTel conventions (directly in Framework or elsewhere) as long as we can provide an upgrade path for existing deployments.

@vmeunier how are you dealing with semantic convention changes? Is this breaking your dashboards? Are you applying filters to normalize meters and traces?