How to evolve stable conventions

[lmolkova]

We should be able to evolve semantic conventions after they become stable or use experimental features in addition to stable semantics.

For example, HTTP spans may use experimental attributes such as http.request.body.size or url.template.

Here's the emerging practice we're using for HTTP:

• Stable instrumentations authored by OpenTelemetry SHOULD NOT produce telemetry that is not described by OpenTelemetry semantic conventions
• explicitly mark any experimental parts as such.
• put experimental attributes into a different (experimental) YAML group and markdown table.
• experimental attributes and/or behavior should be opt-in.
• it's up to instrumentation library to provide an opt-in mechanism (per attribute, feature, or all experimental features).
• after attribute becomes stable, it may be moved to the stable part of the conventions and the requirement level can be changed (as long as it does not introduce breaking changes)

Incompatible changes (definition):

In exceptional cases semantic conventions may introduce incompatible changes to stable parts. For example, HTTP client telemetry did not have a way to record url.template. By allowing to use the attribute (in a rare cases when it's available) on the span name and on metrics, we can significantly improve http client observability

• such changes would require a major version bump for the instrumentation libraries to adopt the new version of semconv.
  • depending on the severity of the change, we may also need to do a major version bump for semantic conventions.
• there should be a very good reason to make such changes: a security bug, a significant net improvement to user experience, etc
• each breaking case should be treated individually and discussed by the semconv and specification communities.

Examples:

• adding url.template opt-in attribute to http.client.request.duration metric is fine

  • it can be added as experimental first
  • instrumentation libraries may add it behind the feature flag
  • eventually it can be moved into a stable metric definition with opt-in requirement level

• adding url.template by default to http.client.request.duration metric is breaking.

  • while the attribute remains experimental it must be opt-in
  • after attribute stabilizes:
    • based on the discussion in semconv community, we may decide that it's a low risk/high reward change and allow populating it by default.
  • in this case, instrumentation libraries may start adding it by default as long as they do a major version bump and document it as a breaking change.

Related issues:

• Should there be any attributes marked as stable inside of experimental conventions? #906
• HTTP client spans: how to represent operation name/route and how to break things #923
• What is the implication of breaking changes to stable semantic conventions #772

[lmolkova]

I believe this is addressed with #1472