Seeking Solutions for an Intermittent Telemetry Logging Issue in .NET 6 with OpenTelemetry: Can We Unearth Stable Alternatives?

[SSPAGNAMN76]

Issue description

In the context of deploying our .NET 6 projects with OpenTelemetry for telemetry logging, we’re encountering an intermittent issue: telemetry logging functions only for a limited span post-deployment before ceasing without error warnings. A memory dump reveals the vanishing of OpenTelemetry instances until a connection pool refresh is initiated. The expectation is to achieve continuous, uninterrupted telemetry logging without the need for pool refreshes.

A key observation is that within our suite, only applications hosted on IIS are affected by this issue. A possible culprit has been identified in the Garbage Collector, which may prematurely deallocate the MeterProvider instance, interrupting logging. Furthermore, potential network-related issues have been highlighted in the logs, yet, given that metrics producer and collector co-exist on the same machine, this is less likely the core of the problem.

Given this scenario:

Environment:

• OpenTelemetry version: 1.6.0
• OS: Windows
• .NET version: .NET Framework 6.0 + Language C#

Attempted Resolutions:

• Adjusted logging levels: setx OTEL_LOG_LEVEL "debug" /M without conclusive results because it is the strategy to have logging when AutoIstrumentation is used,
• Implemented a static field, Provider, retained the MeterProvider instance, and used GC.KeepAlive to prevent early collection by the GC.

Code Configurations:

Below is the configuration class code that serves as our starting point for telemetry logging setup in our applications:

using Microsoft.Extensions.Configuration;

namespace MyProject.Metrics;
public static class MetricsConfigurator
{
    private static readonly AsyncLocal<MetricStreamConfiguration> CurrentCallConfiguration = new();
    private static MeterProvider Provider;

    public static MeterProvider Configure(IConfiguration configuration, Assembly mainAssembly)
    {
        var appName = mainAssembly.GetName();
        var resource = ResourceBuilder.CreateDefault()
            .AddService(serviceName: appName.Name, serviceVersion: appName.Version!.ToString())
            .AddAttributes(new KeyValuePair<string, object>[]
            {
                new("server.name", Environment.MachineName),
                new("process.id", Environment.ProcessId)
            });

        var exportInterval = TimeSpan.FromMilliseconds(configuration.GetValue("OTConf:ExportIntervalMilliseconds", 60000));

        Provider = Sdk.CreateMeterProviderBuilder()
            .AddMeter("*")
            .SetResourceBuilder(resource)
            .AddView(instrument =>
            {
                var config = CurrentCallConfiguration.Value;
                CurrentCallConfiguration.Value = null;
                return config;
            })
            .AddOtlpExporter((eo, mo) =>
            {
                eo.Endpoint = new Uri("http://localhost:5222");
                eo.Protocol = OtlpExportProtocol.Grpc;
                mo.PeriodicExportingMetricReaderOptions = new PeriodicExportingMetricReaderOptions
                {
                    ExportIntervalMilliseconds = (int)exportInterval.TotalMilliseconds
                };
                mo.TemporalityPreference = MetricReaderTemporalityPreference.Delta;
            }).Build();

        return Provider;
    }

    public static void AddOpenTelemetry(this IServiceCollection services, IConfiguration configuration)
    {
        var meterProvider = Configure(configuration, Assembly.GetCallingAssembly());
        GC.KeepAlive(meterProvider);
    }

    public static Meter UsingConfiguration(this Meter meter, MetricStreamConfiguration configuration)
    {
        CurrentCallConfiguration.Value = configuration;
        return meter;
    }
}

Log Results:

Error: [...] Exception: {1}{http://localhost:5110/}{Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: The request was aborted. IOException: The response ended prematurely while waiting for the next frame from the server." [...]

Deep Dive Into IIS Context Issue:

A specific correlation with IIS context has been identified. The issue is reproduced only when the application is executed within IIS and not as a standalone .NET 6 console application.

Final Query

Considering the outlined scenario, attempted resolutions, and the provided code configurations:

• How can the telemetry logging interruption issue be decisively diagnosed and resolved, especially within the IIS hosting context?
• Are there known issues, or suggested explorations, to address potential incongruities/conflicts arising specifically within the IIS context?
• How might we achieve a stable, persistent telemetry logging mechanism in this scenario to avoid future disruptions and ensure reliable logging in production?

We are navigating through a critical path and are eager to resolve this at the earliest. Any insights, recommendations, or similar experiences shared would be highly valuable in crafting a swift, effective resolution. Your expertise and timely response will be immensely appreciated. Thank you for engaging and looking forward to a collaborative discussion.

[SSPAGNAMN76]

Open a bug for developers. Thanks to all for the attention.
#4936