[bug] No metrics emitted after change to 9.0-azurelinux3.0-distroless

[jschweda]

Package

OpenTelemetry.Exporter.OpenTelemetryProtocol

Package Version

Package Name	Version
OpenTelemetry.Extensions.Hosting	1.12.0
OpenTelemetry.Exporter.OpenTelemetryProtocol	1.12.0
OpenTelemetry.Instrumentation.Runtime	1.12.0

Runtime Version

net9.0

Description

We recently changed from mcr.microsoft.com/dotnet/runtime/9.0-azurelinux3.0 to mcr.microsoft.com/dotnet/runtime/9.0-azurelinux3.0-distroless and the metrics export stopped to work.

Metrics were no longer send to the configured endpoint. Metrics are send via grpc.

We tried to activate self-diagnostics like described here but we were not able to see any traces from from OpenTelemetry.

Steps to Reproduce

Change runtime image for application from mcr.microsoft.com/dotnet/runtime/9.0-azurelinux3.0 to mcr.microsoft.com/dotnet/runtime/9.0-azurelinux3.0-distroless

Expected Result

If there is a known issue / missing components in the distroless image I would either expect documentation, a log output in the program or even an error thrown.

Actual Result

Nothing, no metrics send, no traces emitted and no exception thrown.

Additional Context

No response

[martincostello]

We tried to activate self-diagnostics like described here but we were not able to see any traces from from OpenTelemetry.

These diagnostics write a file to the file system, did you check whether the file was created and had any content in it? Asking as "see any traces" suggests you were maybe looking for traces to be emitted.

[jschweda]

There was no file created. Neither found I entries in the traces via dotnet-trace.

[martincostello]

Thanks for the clarification. I'll see if I can repro this myself.

[martincostello]

While I do that, does mcr.microsoft.com/dotnet/runtime/9.0-azurelinux3.0-distroless-extra work?

[marcschier]

Was our first idea as well, i8n missing, but it also did not work for us. Only switching back to non-distroless worked. Not easy to explain though...

[martincostello]

With the app I have to hand which I updated to use mcr.microsoft.com/dotnet/aspnet:9.0.7-azurelinux3.0-distroless tracing worked (which is how I found that SqlClient doesn't work out of the box as DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true and there's no ICU libraries, see dotnet/SqlClient#220).

Are you able to provide a minimal Dockerfile that reproduces the issue?

[jschweda]

@martincostello I'm pretty busy right now. I will provide some more info next week.

[Kielek]

@jschweda, do you have any chance to work on repro steps?

[jschweda]

I'm working on it. but it is not so easy.
I was not capable of creating minimized repo steps. And the image it happens in is pretty stripped down. Did you you use root or the app user in your repo?

[martincostello]

It would have been the app user as that's the default in .NET8 chiselled images (this is the Dockerfile I used)

[jschweda]

We are not using asp. could you please try with this image instead? mcr.microsoft.com/dotnet/runtime/9.0-azurelinux3.0-distroless

Also we switch to an restricted user with

USER $APP_UID

ENTRYPOINT ["dotnet", "your.dll"]