Need clarification on interaction between AddXInstrumentation vs builder.Metrics.[Enable|Disable]Metrics for metrics

[julealgon]

We've been using the OTEL instrumentation packages for over a year now without too much trouble.

However, we recently tried enabling full OTEL in one of our console applications that has hundreds of instances running in production, and this created a problem for us due to the sheer number of metrics being reported to Datadog, chewing up on our custom metric quota and incurring extra costs for us.

Because of this concern, we wanted to disable some/all of the metrics for now until we can come up with a more permanent plan.

On this project, we are using both tracing and metrics instrumentations, configured like this:

builder.Services.AddOpenTelemetry()
    .ConfigureResource(c => c
        .AddHostDetector()
        .AddOperatingSystemDetector()
        .AddProcessDetector()
        .AddProcessRuntimeDetector()
        .AddService("OurAppName"))

    .WithTracing(c => c
        .AddEntityFrameworkCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSqlClientInstrumentation())

    .WithMetrics(c => c
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddProcessInstrumentation())

    .UseOtlpExporter();

Since we want to eventually enable the metrics again over time on a case-by-case basis, I didn't want to just remove the AddXInstrumentation from the metrics section in code. Instead, I wanted a way to turn the metrics off via configuration instead.

This application was recently converted to .NET9, and thus can leverage the new configuration-based metrics setup via the "Metrics" configuration section.

I then proceeded to add this configuration to the project's appsettings.json:

  "Metrics": {
    "EnabledMetrics": {
      "Default": false
    }
  }

According to (the extremely lackluster) documentation, this is supposed to turn off every single metric emitted on the system.

However, we noticed that even after adding these settings, we were still getting metrics pushed like crazy from this console app into our OTEL collector.

I then performed a few tests locally, and I'm now quite confused and would like some clarification.

If I add the Console exporter to the tracing section to see what's happening:

    .WithMetrics(c => c
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddProcessInstrumentation()
        .AddConsoleExporter())

I get basically full metric outputs, regardless of the setting on appsettings.json. Apparently, it is completely ignored.

However, if I instead add the native console output debugger for metrics, using:

builder.Metrics.AddDebugConsole();

I can tell that it does honor my appsettings.json configuration, and nothing is shown when it is set to false, while there is a massive spam of metrics when it is true. This to me is already unexpected.

At this point I got very confused. My understanding was that the AddXInstrumentation calls from the OTEL instrumentation packages were just a "bridge" to the built-in instrumentations in modern .NET versions, and thus I thought that, if I disabled the meters using the native "Metrics" mechanism, that it would also reflect in the OTEL pipeline not seeing them.

But this is clearly not the case, so my understanding is obviously flawed: even if I disable all meters via configuration, apparently the AddXInstrumentation calls "re-enables" them somehow but just for the OTEL pipeline (i.e. no output at all when using Metrics.AddDebugConsole()).

I then performed a final test: I removed all AddXInstrumentation calls from the WithMetrics section of the setup, and suddenly now OTEL appears to properly respect my appsettings.json configuration, although it becomes fully opt-in: if I don't have a "Metrics" config section at all, nothing is emitted. Only when I explicitly set it to true for the Default meter, it enables everything.

This is all super confusing to me. Are we just not supposed to use the AddXInstrumentation methods anymore in modern .NET if we want to be able to dynamically turn meters on and off for Open Telemetry? And if that's the case, would this work for all of the instrumentations I'm using atm?

        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddProcessInstrumentation()

EDIT:

After posting this, I went to check each of the instrumentations above and realized that the Process one does not tap into built-in instrumentations, and instead actually creates the meters/instruments itself. I then realized that the only way to dynamically control that one is to add the instrumentation without adding the meter, which currently requires reflection:

                    .AddInstrumentation(() => Activator.CreateInstance(
                        Type.GetType("OpenTelemetry.Instrumentation.Process.ProcessMetrics, OpenTelemetry.Instrumentation.Process"),
                        nonPublic: true))

And then it works for controlling that instrumentation via "Metrics" configuration, but that's obviously super convoluted and unfriendly.

---

Would this break down completely if I was using an instrumentation package that did not tap into built-in instrumentations, like runtime/process/httpclient seem to do?

I just need some guidance here as all the documentation around this seems incredibly sparse and incomplete, both from the OTEL side as well as from the .NET side itself (with the lack of proper documentation for the "Metrics" config section, which I basically had to reverse engineer from the code to see how it worked...).

Related:

• Adding Trace Sources and Meter Names from Configuration #5529
• [hosting] Support .NET 8 IMetricsBuilder API #4958
• [API Proposal]: Meter Configuration and Pipeline dotnet/runtime#85684