Add exemplar support to Prometheus exporter

[saul]

Fixes #3449
Design discussion issue - not found

Changes

Adds support for exporting histogram exemplars to OpenTelemetry.Exporter.Prometheus.AspNetCore and OpenTelemetry.Exporter.Prometheus.HttpListener. There are a couple of aspects that I'd appreciate some guidance on:

• The spec states "If no exemplars exist on a bucket, the highest exemplar from a lower bucket MUST be used, even though it is a duplicate of another bucket’s exemplar" but my implementation doesn't follow this. I'm wondering what the rationale is behind this as for large histograms I can imagine this would produce a lot of extraneous data.
• The spec states "For Prometheus pull endpoints, only a single exemplar is able to be added to each bucket, so the largest exemplar from each bucket MUST be used, if attaching exemplars" but exemplars as implemented in OTel.Net do NOT preserve the largest exemplar in each bucket on histograms. Simply the most recent value for that bucket is stored as the exemplar, regardless of whether it is larger than the current exemplar or not
• Do the exemplars need to be 'wiped' between scrapes of the /metrics endpoint? Or should the exemplars be persistent in that once a bucket has an exemplar, it will always be returned by the /metrics endpoint until it is replaced by another exemplar.
• Is this considered an API-breaking change? I wonder if it would be best to add a bool option to PrometheusAspNetCoreOptions to enable exemplars, with the default being disabled (i.e. current behaviour)
  • "Prometheus Exporters are not stable yet, so no concern about breaking changes here." @cijothomas

Merge requirement checklist

• CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
• Unit tests added/updated
• Appropriate CHANGELOG.md files updated for non-trivial changes
• Changes in public API reviewed (if applicable) - see question above re adding a PrometheusAspNetCoreOptions option

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: cijothomas / name: Cijo Thomas (4261b1d)
• ✅ login: Kielek / name: Piotr Kiełkowicz (bb24629, e17a2af, bf24a25, 0b0e001, 725a730, 86cb1c7)
• ✅ login: saul / name: Saul Rennison (606b69e, f336968, 9cac3e7, 8550c31, 24b2c6a)

[cijothomas]

Is this considered an API-breaking change? I wonder if it would be best to add a bool option to PrometheusAspNetCoreOptions to enable exemplars, with the default being disabled (i.e. current behaviour)

I think you can enable it by default, given Exemplars itself is off-by-default in the SDK! So if prometheus exporter is exporting exemplars, users has already made an explicit decision to enable it at the sdk.
Another thing - Prometheus Exporters are not stable yet, so no concern about breaking changes here.

[cijothomas]

• ❌ - login: @saul / name: Saul Rennison . The commit (fe78b00, a609c15) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

@saul Can you sign the CLA part, as it is a mandatory pre-req for accepting any contributions to OpenTelemetry. If you are working on behalf on an employer, you may want to consult your employer before CLA. (Note: This is a one time requirement only)

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[saul]

Hi @cijothomas the CLA has been signed and this is ready for review when you have time. Thanks

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[cijothomas]

@robertcoltheart Tagging you here, to see if you have some bandwidth to help review this PR, given you have made several contributions to the Prometheus Exporter!

[cijothomas]

The spec states "If no exemplars exist on a bucket, the highest exemplar from a lower bucket MUST be used, even though it is a duplicate of another bucket’s exemplar" but my implementation doesn't follow this. I'm wondering what the rationale is behind this as for large histograms I can imagine this would produce a lot of extraneous data.

I am not sure of the reasoning for this either...You can put a TODO in the code and state this is not implemented, and we can follow up with the spec.

[saul]

I believe I've fixed the failing test now.

Unfortunately when I push, my commits don't seem to trigger CI - do I need to manually trigger CI?

[Kielek]

Unfortunately when I push, my commits don't seem to trigger CI - do I need to manually trigger CI?

First contribution requires approval to execute pipeliene (until PR is merged). Any furter changes will trigger pipeline automatically.

CI triggered.

[saul]

Thanks for your help @Kielek. From what I can see all of the checks have passed now.

[Kielek]

@robertcoltheart, do you have some time to make full review of this PR?

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.36%. Comparing base (71abd41) to head (e17a2af).
Report is 28 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5929      +/-   ##
==========================================
- Coverage   86.36%   86.36%   -0.01%     
==========================================
  Files         257      257              
  Lines       11584    11581       -3     
==========================================
- Hits        10005    10002       -3     
  Misses       1579     1579              

Files with missing lines	Coverage Δ
...Listener/Internal/PrometheusExporterEventSource.cs	66.66% <ø> (ø)
...s.HttpListener/Internal/PrometheusSerializerExt.cs	100.00% <ø> (ø)

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[saul]

Hi team - is someone able to take a look at this? This PR's been open for over 3 months now. Thank you

[Kielek]

@saul, sorry for delay with reviewing. I am pretty busy with other work.

Lets start with this

The spec states "If no exemplars exist on a bucket, the highest exemplar from a lower bucket MUST be used, even though it is a duplicate of another bucket’s exemplar" but my implementation doesn't follow this. I'm wondering what the rationale is behind this as for large histograms I can imagine this would produce a lot of extraneous data.

If there is hard requirement from the spec, we need to follow. As https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#:~:text=If%20no%20exemplars%20exist%20on,trace%20and%20span%20IDs%2C%20respectively. is still marked as Development, you can try to create otel-spec issue to modify this. For now, it should be hard requirement before merge.

[Kielek]

Do the exemplars need to be 'wiped' between scrapes of the /metrics endpoint? Or should the exemplars be persistent in that once a bucket has an exemplar, it will always be returned by the /metrics endpoint until it is replaced by another exemplar.

I do not found anything in specification. I would recommend to create issue under https://github.com/open-telemetry/opentelemetry-specification to clarify. There is a chance that there will be a need to modify some documentation there.

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.