Why is /metrics showing up in my dashboard?

[leandrobattochio]

So I just added OpenTelemetry to my API, configured prometheus and grafana and when using the .NET Grafana dashboards I can see that calls to /metrics are polluting all my charts.

I ended up editing the dashboard json to exclude all routes that contained "/metrics". Why is that endpoint showing up in the dashboard? It does not make sense to me. I don't care about that endpoint. Is there a way to configure it so it won't even show in the data prometheus gather or I really have to filter the dashboard?

Maybe I configured the OpenTelemetry wrong in my API?

    builder.Services.AddOpenTelemetry()
        .WithMetrics(x =>
        {
            x.AddPrometheusExporter();

            x.AddMeter("Microsoft.AspNetCore.Hosting", "Microsoft.AspNetCore.Server.Kestrel");
        });

    var app = builder.Build();
    app.MapPrometheusScrapingEndpoint();

[davidfowl]

@reyang Is there a way to opt requests out of telemetry exporting?

[reyang]

Yes. OpenTelemetry .NET has a contract called SuppressInstrumentation, here is an example https://github.com/open-telemetry/opentelemetry-dotnet/blob/0b9cf42e9af668fe6d3af86254f9758c7c821f5e/docs/metrics/extending-the-sdk/MyExporter.cs#L21. It requires collaboration between two components:

1. Exporter needs to set SuppressInstrumentation, so that any RPC triggered by the exporter have the contextual flag.
2. Instrumentation Library / Instrumented Library needs to access the SuppressInstrumentation flag, and if the flag is set, avoid calling the telemetry API (instrumentation/instrument library owns the decision whether to suppress instrumentation or not).

[leandrobattochio]

I'm sorry but I can't seem to understand how it is supposed to be done. Can you provide an example code?

[reyang]

@leandrobattochio it requires work to be done on the instrumentation side (e.g. ASP.NET Core in this case) before you can use it. open-telemetry/opentelemetry-dotnet#960 has some additional context (primarily targeting SDK authors and instrumentation authors, not the user).

[leandrobattochio]

Well, so I just got a workaround. By editing Prometheus.yml file, add the following config in the target:

`
metric_relabel_configs:

  - source_labels: [http_route]
    regex: '/metrics'
    action: drop

`

And there you go, endpoint is gone from metrics.

[raffaeler]

I am also trying to avoid to display from the Aspire dashboard the Prometheus requests over /metrics but the following prometheus.yaml changes didn't help:

Suggestions?

global:
  scrape_interval: 10s # relaxing the scrape interval

scrape_configs:
  - job_name: 'copilot'
    #metrics_path: /metrics/azureV2
    static_configs:
      # same port as in launchSettings.json (AppHost/http)
      - targets: [
        # various endpoints here
        ]
    metric_relabel_configs:
      - source_labels: [http_route]
        regex: '/metrics'
        action: drop