OpenSSL Server Verification Fails on .NET Jammy Chiseled containers

[dlinkov-roblox]

Hello!

We've been pushing for adoption of the chiseled containers, but have run into some difficulties with the profiler we use. We've tracked the issue down to the underlying HTTP library in use here. In particular, we are hitting one of the three error cases starting here based on the error logged by the profiler when it tries to export data. The exact same set up works in the equivalent jammy Dockerfile (dotnet/aspnet:6.0-jammy vs dotnet/aspnet:6.0-jammy-chiseled).

We've tried to copy all of the libraries from a builder stage using the dotnet/aspnet:6.0-jammy image.

COPY --from=builder /usr/bin/openssl /usr/bin/openssl
COPY --from=builder /usr/lib/ /usr/lib/

This leads us to think this is not a library issue (which had been our hunch). We've also tried to (with no success) change ownership of certain .so files thinking that might be the issue.

Are there any other significant changes between the jammy image and the chiseled jammy image that could lead to this mismatch in behaviour?

Thanks in advance!

[richlander]

Can you try running the regular jammy image as non-root? That would help rule out some of the permissions issues.

Do you need a different OpenSSL than the one we include?

[jaskaransinghdr6j]

Gotcha, yes we'll remove openssl and and dependencies we were trying to copy explicitly when trying to chisel this package ourself.

[cjdcordeiro]

Hi! As an additional check, just to be sure, can you see if the issue is somehow related to ca-certificates? I.e. the current slices for ca-certificates aren't running update-ca-certificates (atm) after the slice installation.

[richlander]

If you are using Pyroscope, I'm curious why your COPY lines don't look more like: https://hub.docker.com/r/pyroscope/pyroscope-dotnet. Is it the case that they didn't work and the two lines you shared are workarounds?

[jaskaransinghdr6j]

We were able to resolve this by copying over raw /usr/share/ca-certificates/ from the builder. We used to run update-ca-certificates in the builder (jammy) and then copy over everything in '/etc/ssl/certs' which includes the ca-certificate.crt file. For some reason in this case, the underlying SSL implementation of the httplib also wanted the raw .pem files of the root CAs (we still dont know why)

tldr;

In the jammy builder:

RUN update-ca-certificates

In the runtime chiseled image:

COPY --from=builder /usr/share/ca-certificates/ /usr/share/ca-certificates/
COPY --from=builder /usr/local/share/ca-certificates/ /usr/local/share/ca-certificates/
COPY --from=builder /etc/ssl/certs /etc/ssl/certs

[jaskaransinghdr6j]

This was debugged by looking at STRACE logs to understand which file descriptors are being called in openat() syscalls.

It's still beyond our understanding why raw cets would still be required even after running update-ca-certificates and copying the generated concatenated cert to /etc/ssl/certs

[richlander]

That's a very useful finding. This challenge has come up a few times. Your pattern looks good and that means you can continue using the official images we publish.

We made a similar change for Alpine: https://learn.microsoft.com/dotnet/core/compatibility/containers/8.0/ca-certificates-package

I take it that you are good to go now?

[jaskaransinghdr6j]

Thanks for your help @richlander! I think we can close this @dlinkov-roblox

[dlinkov-roblox]

Thanks for the help @richlander, this was exceptional support and we really appreciate it!

[richlander]

My pleasure. I'm very motivated to help teams adopt chiseled images. We've put a lot of effort into them and think that this form factor is the future. Don't hesitate to reach out for help again.