Refactor Dockerfile

[Raimo33]

This PR refactors the dockerfile according to issue #8409

changes:

• remove useless python-builder stage
• verify GPG keys (Docker: verify GPG keys #8405)
• only include the necessary binaries in final image
• use of ARG instead of ENV for build-time vars
• remove useless apt packages
• use of ADD instead of curl
• set global SHELL
• reduce redundancy of multi-arch image selections
• use of tee + heredocs instead of messy echo's
• stripped compiled binaries for smaller image
• remove manual tini in favor of docker --init flag
• avoid compiling sqlite, zlib, postgres from source
• replace poetry with uv
• avoid downloading all history/branches for submodules
• parallelize make and submodules cloning
• better docker caching of layers
• fix CPU compatibility bug (Dockerfile is not CPU agnostic #8456)
• avoid running cargo with QEMU. true cross compilation.
• avoid installing clippy and rust-doc components
• any host to any target image building (not tested)
• avoid installing manpages
• update base image from bookworm to trixie
• fix poetry trying to install the current project as standalone package (crashed the build before)
• fix armhf target (crashed the build before)

Feature	Before	After	Improvement
Build Time (amd64 to arm64)	90.6 min	10.2 min	8.8x
Image Size (amd64)	512 MB	247 MB	2.07x
Lines of Code	316	199	1.6x

build time was benchmarked on a 12th Gen Intel(R) Core(TM) i5-1235U and a 20 Mbps connection

[ShahanaFarooqui]

@Raimo33 As per @cdecker's suggestion here, we should modify the Dockerfile to run as a non-root user for improved security. Since we're already refactoring the Dockerfile, could we incorporate this change as well?

[Raimo33]

make install currently calls git describe to know the version of the program. This means that the .git/ folder needs to be copied into the Dockerfile for it to work. But copying the .git/ folder almost always invalidates the cache of the COPY . . command.

I'd like to explore ways to not include the .git/ folder (aka placing it in the .dockerignore). Maybe we can add an option to the Makefile that doesn't call git describe

[cdecker]

Nice changes, and with the switch to uv things may get even a bit simpler still. Undraft once ready, and we'll review again and merge asap 👍

[Raimo33]

@cdecker Adding a non-root USER directly inside the Dockerfile is not a reliable option when using mounted volumes. The reason is that file permissions on mounted volumes are determined by the UID and GID of the host system, not the container. Hardcoding a UID, for example USER 1000:1000, might work in some environments, but it is not guaranteed to match the host system’s user. On many systems, the first regular user is UID 1000, but this is not universal, and in environments like CI/CD runners, containers, or systems with different user configurations, the UID/GID may differ.

Hardcoding the UID inside the Dockerfile creates a fragile assumption and can easily lead to permission issues. Some repos such as mempool do just that, but I don't deem it robust.

I have fought with docker user permissions for years, and when there are volumes involved, unless using docker-compose, the best thing is letting the user set the user at runtime via docker run --user instead of hardcoding it in the Dockerfile. But apparently lightningd needs root access to create the data directory, so the only option is running as root.

Could not make directory /.lightning: Permission denied

[Raimo33]

The previous Dockerfile was not cross-compiling rust packages. Cargo was emulating with QEMU instead of cross compiling, despite the efforts as seen by modifying the cargo config.toml. I've now fixed it and it correctly cross compiles (much faster than QEMU), but the Makefile needs a retouch as it is searching the binaries in the wrong folder.

#53 185.3 cp: cannot stat 'target/release/cln-lsps-service': No such file or directory

[Raimo33]

Closes #8409 #8405 #8456 #5655

[king-11]

can't we add it in dockerignore?

[Raimo33]

no. because make needs it. it calls git describe to get the tag