gcr.io/distroless/cc-debian12:nonroot huge performance decrease for rust based app

[soundofspace]

I recently switched a Dockerfile that builds a Rama derivative from:

FROM debian:bookworm-slim AS prod

RUN apt-get update && \
    apt-get install --no-install-recommends -y ca-certificates

to this:

FROM gcr.io/distroless/cc-debian12:nonroot AS prod

And saw a huge decrease in performance of about 4x and what looks like a memory leak (gets way worse the longer it runs). Deployed 11:15, reverted 12:05. I later did this again just to be sure and noticed the exact same results

I'm trying to diagnose what could be causing this, but so far have not found anything, any help or pointers would be greatly appreciated.

What has been checked:

• All versions in dockerfile are using debian 12: rust:1.85.0-bookworm to build and then debian:bookworm-slim or gcr.io/distroless/cc-debian12:nonroot
• Check ldd what is dynamically linked:

        linux-vdso.so.1 (0x00007ffe4f7d8000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f14d629d000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f14d61b6000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f14d5f8d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f14d8a96000)

Distroless:

~ $ sha1sum /lib/x86_64-linux-gnu/libgcc_s.so.1
ccf733350d9c59a03a6427b698c5301016c6ea43  /lib/x86_64-linux-gnu/libgcc_s.so.1
~ $ sha1sum /lib/x86_64-linux-gnu/libm.so.6
c5f419636f43aa556f57ae22b161c4a21b6b383a  /lib/x86_64-linux-gnu/libm.so.6
~ $ sha1sum /lib/x86_64-linux-gnu/libc.so.6
05f456ef56882dd764b69813e3a0bf5b6a4fdc54  /lib/x86_64-linux-gnu/libc.so.6
~ $ sha1sum /lib64/ld-linux-x86-64.so.2
edcca2aee14c3e505e0f7b56eb35210cc2cceea0  /lib64/ld-linux-x86-64.so.2

Debian:

root@debian-full:/# sha1sum /lib/x86_64-linux-gnu/libgcc_s.so.1
ccf733350d9c59a03a6427b698c5301016c6ea43  /lib/x86_64-linux-gnu/libgcc_s.so.1
root@debian-full:/# sha1sum /lib/x86_64-linux-gnu/libm.so.6
c5f419636f43aa556f57ae22b161c4a21b6b383a  /lib/x86_64-linux-gnu/libm.so.6
root@debian-full:/#  sha1sum /lib/x86_64-linux-gnu/libc.so.6
05f456ef56882dd764b69813e3a0bf5b6a4fdc54  /lib/x86_64-linux-gnu/libc.so.6
root@debian-full:/# sha1sum /lib64/ld-linux-x86-64.so.2
edcca2aee14c3e505e0f7b56eb35210cc2cceea0  /lib64/ld-linux-x86-64.so.2

Rama is a proxy service that internally uses a slightly modified Hyper and Boringssl, and quite a lot more but all those should be statically linked and switching to distroless should not affect that.

We are also using Jemalloc as the allocator. I didn't find any known issues with Jemalloc on distroless, but seeing the huge increase in memory usage, it seems like a potential culprit.

Are there any other things I can investigate, or does anyone have a clue on what could be going wrong here?

[reneleonhardt]

Oh my god, could you build again with rust:1.87.0-bookworm?
They said LLVM 20 could bring some performance:
https://blog.rust-lang.org/2025/05/15/Rust-1.87.0/

A quick search only revealed one distroless + memory leak issue:
hyperium/hyper#3130