Merge bitcoin/bitcoin#25490: guix: more cross arch reproducibility (x86_64 -> arm64)

laanwj · laanwj · commit b04f42efe31e · 2022-07-01T11:27:17.000+02:00
54faac9 guix: Remove guix store paths from glibc (Andrew Chow) 1d4d711 guix: Map all guix store prefixes to /usr (Andrew Chow) Pull request description: This cherry-picks two commits from #24615, with minor edits. When building master, only the `x86_64-apple-darwin` build is reproducible across x86_64 and arm64. With these two changes we get x86_64 -> arm64 reproducibility for: * `powerpc64-linux-gnu` * `powerpc64le-linux-gnu` * `x86_64-apple-darwin` * `x86_64-linux-gnu` We can't compare `aarch64-linux-gnu`, because we can't currently build for `aarch64-linux-gnu` on `aarch64-linux-gnu`/`arm64`. See (#22458). For all the other hosts, the reproducibility issues are known / being worked on (see discussion in #24615). However there's no real reason to wait for those to be fixed before merging these changes, as it'd be great to have cross arch reproducibility just for `x86_64-linux-gnu`. I'm also unsure about the approach taken in that PR in regards to the libtool changes (and think there might even be a Guix bug involved). As I've added to the patch file, we may be able to drop these patches and use `--with-nonshared-cflags` when we are building newer versions of glibc. Guix Build (x86_64): ```bash 1eadc59775a209e707539a6bdfe4c96e13a17f5373bfaf6477a65c1a44b2459d guix-build-54faac968971/output/arm-linux-gnueabihf/SHA256SUMS.part 1f424e448223a1e1ee1658efeb3bcc0d8b08a2a2bdc9d2dc779c931b956b527f guix-build-54faac968971/output/arm-linux-gnueabihf/bitcoin-54faac968971-arm-linux-gnueabihf-debug.tar.gz 212f55cf55dac34a3a6471bf0585f5ba1ed0a4801e9c0003961617881edb7ee7 guix-build-54faac968971/output/arm-linux-gnueabihf/bitcoin-54faac968971-arm-linux-gnueabihf.tar.gz f920fdc9407371be8fc5638a0f5ce2f1202889d820bda4451096d162b5d28d94 guix-build-54faac968971/output/arm64-apple-darwin/SHA256SUMS.part 066098fe095a3dd46fa3c84e459aca1e8e4b3d564ab3f20a2542c8c46dd37b5a guix-build-54faac968971/output/arm64-apple-darwin/bitcoin-54faac968971-arm64-apple-darwin-unsigned.dmg d2dc320975f8ec67595c274fcb7eda56c18ba72b905160537be97408cfdf1baf guix-build-54faac968971/output/arm64-apple-darwin/bitcoin-54faac968971-arm64-apple-darwin-unsigned.tar.gz 68013ac011ebac25e8ee2d2421266c66d1e20d309133eb6121ba89807e17fda5 guix-build-54faac968971/output/arm64-apple-darwin/bitcoin-54faac968971-arm64-apple-darwin.tar.gz d992dd9f50fec89e0bdf63e8e9352ebcd2b503424cb441f01c06074aa0b63b0e guix-build-54faac968971/output/dist-archive/bitcoin-54faac968971.tar.gz e26ea298d15a87ce484afa59ca0c36ecfd173314b440e091cd93de11fe2dd91c guix-build-54faac968971/output/powerpc64-linux-gnu/SHA256SUMS.part 1a17091e7e515cc89a2e0a91a08ea0deb48847d5650be936f3365449002a59a5 guix-build-54faac968971/output/powerpc64-linux-gnu/bitcoin-54faac968971-powerpc64-linux-gnu-debug.tar.gz 63ce21f6ee3a971a7a4533934ec559c727d2e3fc123fb65b2dec622168af76d5 guix-build-54faac968971/output/powerpc64-linux-gnu/bitcoin-54faac968971-powerpc64-linux-gnu.tar.gz 267ff9d9ed3d8108033d676218399c4e56ba83eb30a3b4a6417cac8d1ce5e2f4 guix-build-54faac968971/output/powerpc64le-linux-gnu/SHA256SUMS.part a0db19b9829bd65067e2075c2d3e55065d03a4456a31ccb3ed8a70f7f3dcf1a0 guix-build-54faac968971/output/powerpc64le-linux-gnu/bitcoin-54faac968971-powerpc64le-linux-gnu-debug.tar.gz 9971ba819854a77306358b31ddc2d21074b8cca57cfd8ce7f2ca83a1cc8f0a46 guix-build-54faac968971/output/powerpc64le-linux-gnu/bitcoin-54faac968971-powerpc64le-linux-gnu.tar.gz e18fe6d4db3048368db25846555a30ec97f71dcfdfcc3e86d7fe46c33c762a26 guix-build-54faac968971/output/riscv64-linux-gnu/SHA256SUMS.part 4879fd332bee8570e6edfe5d0cbd3eea7df18f058ed25286aedb377858e27793 guix-build-54faac968971/output/riscv64-linux-gnu/bitcoin-54faac968971-riscv64-linux-gnu-debug.tar.gz 470fafbab9d7328e172c658f58fd6f62f7890ce5d3dfe3bbfacd717b6d2c00f3 guix-build-54faac968971/output/riscv64-linux-gnu/bitcoin-54faac968971-riscv64-linux-gnu.tar.gz 6b119a38b8ac1dd50e3cee69e8464ab21d624217cb3f50a1aef216e9ece27946 guix-build-54faac968971/output/x86_64-apple-darwin/SHA256SUMS.part afff2a8184007d6a3eaf7b6735417e7b9b404a801306c71f7d653907055d442c guix-build-54faac968971/output/x86_64-apple-darwin/bitcoin-54faac968971-x86_64-apple-darwin-unsigned.dmg 775298169fa45197f5b560e5581d6e3c021ed009487065f7e3a042914396b3cd guix-build-54faac968971/output/x86_64-apple-darwin/bitcoin-54faac968971-x86_64-apple-darwin-unsigned.tar.gz 51b0d2adae63aeca021621ec25278c202e82d8bc3ef5e7a9d2bfd93bd7a3fcab guix-build-54faac968971/output/x86_64-apple-darwin/bitcoin-54faac968971-x86_64-apple-darwin.tar.gz ca9cd0083083ec1632a37396076fef638c00fed99efbb9e39521195867746301 guix-build-54faac968971/output/x86_64-linux-gnu/SHA256SUMS.part 0e5a017bca3c0a2a466d230a470ec9c44b5d0bfd60d98f62a3a54f4cccb8cd23 guix-build-54faac968971/output/x86_64-linux-gnu/bitcoin-54faac968971-x86_64-linux-gnu-debug.tar.gz 3b26566a553f17ccbcbf548675d0e8ce47a58a1e23c6b7c9792565117a223855 guix-build-54faac968971/output/x86_64-linux-gnu/bitcoin-54faac968971-x86_64-linux-gnu.tar.gz fa2f240f7bd7cfadfeb08188c044c8e34fd9c24785f4e2035c1256c3173c5589 guix-build-54faac968971/output/x86_64-w64-mingw32/SHA256SUMS.part 0212ae95d100c9a4dbe062a14b49952279c91cbf352c786369320c3ed006c23a guix-build-54faac968971/output/x86_64-w64-mingw32/bitcoin-54faac968971-win64-debug.zip 750aa7c31cfa1bd5e0ae3f2ea52e526a73f1d3879b9f1a365967bbc317d4cca4 guix-build-54faac968971/output/x86_64-w64-mingw32/bitcoin-54faac968971-win64-setup-unsigned.exe f7bdda020d299df778fd68ad556d8124d87f7f9f0c4a77b62e05db20f5bbec2b guix-build-54faac968971/output/x86_64-w64-mingw32/bitcoin-54faac968971-win64-unsigned.tar.gz d7f8b16634ceb95db3d28a024827a51f689dc0ab34ed40f205861bbc254f6206 guix-build-54faac968971/output/x86_64-w64-mingw32/bitcoin-54faac968971-win64.zip ``` Guix Build (arm64): ```bash e4492294c284054e8378f8ac0c08b5d2efe5eeeec57ca12c3192da87a4dd4266 guix-build-54faac968971/output/arm-linux-gnueabihf/SHA256SUMS.part 1f424e448223a1e1ee1658efeb3bcc0d8b08a2a2bdc9d2dc779c931b956b527f guix-build-54faac968971/output/arm-linux-gnueabihf/bitcoin-54faac968971-arm-linux-gnueabihf-debug.tar.gz b2c454f589afea22d5126be14d1b9eb3aed7d99d59e50591db18c2cc3f190b23 guix-build-54faac968971/output/arm-linux-gnueabihf/bitcoin-54faac968971-arm-linux-gnueabihf.tar.gz 575bbf78dd6002a7a5653277c24cd8d98471b6454e0801b558ac2a754788f2fa guix-build-54faac968971/output/arm64-apple-darwin/SHA256SUMS.part 03d8175e0db26b56fee757b307b8b519e4df0c639caa6afdfdea5ce7de63ecc4 guix-build-54faac968971/output/arm64-apple-darwin/bitcoin-54faac968971-arm64-apple-darwin-unsigned.dmg 8ab4e3a65f09168b42c40ec736e507eb3e9b787d27a56db21a3c601e81d81262 guix-build-54faac968971/output/arm64-apple-darwin/bitcoin-54faac968971-arm64-apple-darwin-unsigned.tar.gz 43024487cf0120a0a42aeba228f01f2a5303d2940d838f59106fa246a484662d guix-build-54faac968971/output/arm64-apple-darwin/bitcoin-54faac968971-arm64-apple-darwin.tar.gz d992dd9f50fec89e0bdf63e8e9352ebcd2b503424cb441f01c06074aa0b63b0e guix-build-54faac968971/output/dist-archive/bitcoin-54faac968971.tar.gz e26ea298d15a87ce484afa59ca0c36ecfd173314b440e091cd93de11fe2dd91c guix-build-54faac968971/output/powerpc64-linux-gnu/SHA256SUMS.part 1a17091e7e515cc89a2e0a91a08ea0deb48847d5650be936f3365449002a59a5 guix-build-54faac968971/output/powerpc64-linux-gnu/bitcoin-54faac968971-powerpc64-linux-gnu-debug.tar.gz 63ce21f6ee3a971a7a4533934ec559c727d2e3fc123fb65b2dec622168af76d5 guix-build-54faac968971/output/powerpc64-linux-gnu/bitcoin-54faac968971-powerpc64-linux-gnu.tar.gz 267ff9d9ed3d8108033d676218399c4e56ba83eb30a3b4a6417cac8d1ce5e2f4 guix-build-54faac968971/output/powerpc64le-linux-gnu/SHA256SUMS.part a0db19b9829bd65067e2075c2d3e55065d03a4456a31ccb3ed8a70f7f3dcf1a0 guix-build-54faac968971/output/powerpc64le-linux-gnu/bitcoin-54faac968971-powerpc64le-linux-gnu-debug.tar.gz 9971ba819854a77306358b31ddc2d21074b8cca57cfd8ce7f2ca83a1cc8f0a46 guix-build-54faac968971/output/powerpc64le-linux-gnu/bitcoin-54faac968971-powerpc64le-linux-gnu.tar.gz 603e09db23e20ee834cfdeaa58517fa6795779131f76c8e67c79ee4118043ba6 guix-build-54faac968971/output/riscv64-linux-gnu/SHA256SUMS.part 4879fd332bee8570e6edfe5d0cbd3eea7df18f058ed25286aedb377858e27793 guix-build-54faac968971/output/riscv64-linux-gnu/bitcoin-54faac968971-riscv64-linux-gnu-debug.tar.gz aad8eb83730dbf079ee2c894e33ebf6df78e302500493b10e72a2aab9fa51b49 guix-build-54faac968971/output/riscv64-linux-gnu/bitcoin-54faac968971-riscv64-linux-gnu.tar.gz 6b119a38b8ac1dd50e3cee69e8464ab21d624217cb3f50a1aef216e9ece27946 guix-build-54faac968971/output/x86_64-apple-darwin/SHA256SUMS.part afff2a8184007d6a3eaf7b6735417e7b9b404a801306c71f7d653907055d442c guix-build-54faac968971/output/x86_64-apple-darwin/bitcoin-54faac968971-x86_64-apple-darwin-unsigned.dmg 775298169fa45197f5b560e5581d6e3c021ed009487065f7e3a042914396b3cd guix-build-54faac968971/output/x86_64-apple-darwin/bitcoin-54faac968971-x86_64-apple-darwin-unsigned.tar.gz 51b0d2adae63aeca021621ec25278c202e82d8bc3ef5e7a9d2bfd93bd7a3fcab guix-build-54faac968971/output/x86_64-apple-darwin/bitcoin-54faac968971-x86_64-apple-darwin.tar.gz ca9cd0083083ec1632a37396076fef638c00fed99efbb9e39521195867746301 guix-build-54faac968971/output/x86_64-linux-gnu/SHA256SUMS.part 0e5a017bca3c0a2a466d230a470ec9c44b5d0bfd60d98f62a3a54f4cccb8cd23 guix-build-54faac968971/output/x86_64-linux-gnu/bitcoin-54faac968971-x86_64-linux-gnu-debug.tar.gz 3b26566a553f17ccbcbf548675d0e8ce47a58a1e23c6b7c9792565117a223855 guix-build-54faac968971/output/x86_64-linux-gnu/bitcoin-54faac968971-x86_64-linux-gnu.tar.gz da10e58c2616b7d96fb00d24f34834b737b190c91bd533fc11130c5faf77d697 guix-build-54faac968971/output/x86_64-w64-mingw32/SHA256SUMS.part 7b69ac09edb7795b61242d8b929808b7615e4011f1d312d495cc9410ded6c574 guix-build-54faac968971/output/x86_64-w64-mingw32/bitcoin-54faac968971-win64-debug.zip 750aa7c31cfa1bd5e0ae3f2ea52e526a73f1d3879b9f1a365967bbc317d4cca4 guix-build-54faac968971/output/x86_64-w64-mingw32/bitcoin-54faac968971-win64-setup-unsigned.exe f7bdda020d299df778fd68ad556d8124d87f7f9f0c4a77b62e05db20f5bbec2b guix-build-54faac968971/output/x86_64-w64-mingw32/bitcoin-54faac968971-win64-unsigned.tar.gz e90e9a7e86839cbbde7b17610b7fb6eb9a960703232d3b92040dce8df55861bd guix-build-54faac968971/output/x86_64-w64-mingw32/bitcoin-54faac968971-win64.zip ``` ACKs for top commit: laanwj: ACK 54faac9 Tree-SHA512: 258e7de8e8df00995b6e952ef10d054ad127237265811eaafa537501be84459ae8f8f638618365d81e4eee8b7013db768b61c343d68e46d1d90a194bdc26b852
diff --git a/contrib/guix/libexec/build.sh b/contrib/guix/libexec/build.sh
@@ -223,6 +223,7 @@ CONFIGFLAGS="--enable-reduce-exports --disable-bench --disable-gui-tests --disab
 
 # CFLAGS
 HOST_CFLAGS="-O2 -g"
+HOST_CFLAGS+=$(find /gnu/store -maxdepth 1 -mindepth 1 -type d -exec echo -n " -ffile-prefix-map={}=/usr" \;)
 case "$HOST" in
     *linux*)  HOST_CFLAGS+=" -ffile-prefix-map=${PWD}=." ;;
     *mingw*)  HOST_CFLAGS+=" -fno-ident" ;;
diff --git a/contrib/guix/manifest.scm b/contrib/guix/manifest.scm
@@ -535,7 +535,8 @@ inspecting signatures in Mach-O binaries.")
                                            "glibc-versioned-locpath.patch"
                                            "glibc-2.24-elfm-loadaddr-dynamic-rewrite.patch"
                                            "glibc-2.24-no-build-time-cxx-header-run.patch"
-                                           "glibc-2.24-fcommon.patch"))))))
+                                           "glibc-2.24-fcommon.patch"
+                                           "glibc-2.24-guix-prefix.patch"))))))
 
 (define-public glibc-2.27/bitcoin-patched
   (package
@@ -552,7 +553,8 @@ inspecting signatures in Mach-O binaries.")
                 "1b2n1gxv9f4fd5yy68qjbnarhf8mf4vmlxk10i3328c1w5pmp0ca"))
               (patches (search-our-patches "glibc-ldd-x86_64.patch"
                                            "glibc-2.27-riscv64-Use-__has_include-to-include-asm-syscalls.h.patch"
-                                           "glibc-2.27-dont-redefine-nss-database.patch"))))))
+                                           "glibc-2.27-dont-redefine-nss-database.patch"
+                                           "glibc-2.27-guix-prefix.patch"))))))
 
 (packages->manifest
  (append
diff --git a/contrib/guix/patches/glibc-2.24-guix-prefix.patch b/contrib/guix/patches/glibc-2.24-guix-prefix.patch
@@ -0,0 +1,25 @@
+Without ffile-prefix-map, the debug symbols will contain paths for the
+guix store which will include the hashes of each package. However, the
+hash for the same package will differ when on different architectures.
+In order to be reproducible regardless of the architecture used to build
+the package, map all guix store prefixes to something fixed, e.g. /usr.
+
+We might be able to drop this in favour of using --with-nonshared-cflags
+when we being using newer versions of glibc.
+
+--- a/Makeconfig
++++ b/Makeconfig
+@@ -950,6 +950,10 @@ object-suffixes-for-libc += .oS
+ # shared objects.  We don't want to use CFLAGS-os because users may, for
+ # example, make that processor-specific.
+ CFLAGS-.oS = $(CFLAGS-.o) $(PIC-ccflag)
++
++# Map Guix store paths to /usr
++CFLAGS-.oS += `find /gnu/store -maxdepth 1 -mindepth 1 -type d -exec echo -n " -fdebug-prefix-map={}=/usr" \;`
++
+ CPPFLAGS-.oS = $(CPPFLAGS-.o) -DPIC -DLIBC_NONSHARED=1
+ libtype.oS = lib%_nonshared.a
+ endif
+-- 
+2.35.1
+
diff --git a/contrib/guix/patches/glibc-2.27-guix-prefix.patch b/contrib/guix/patches/glibc-2.27-guix-prefix.patch
@@ -0,0 +1,25 @@
+Without ffile-prefix-map, the debug symbols will contain paths for the
+guix store which will include the hashes of each package. However, the
+hash for the same package will differ when on different architectures.
+In order to be reproducible regardless of the architecture used to build
+the package, map all guix store prefixes to something fixed, e.g. /usr.
+
+We might be able to drop this in favour of using --with-nonshared-cflags
+when we being using newer versions of glibc.
+
+--- a/Makeconfig
++++ b/Makeconfig
+@@ -992,6 +992,10 @@ object-suffixes :=
+ CPPFLAGS-.o = $(pic-default)
+ # libc.a must be compiled with -fPIE/-fpie for static PIE.
+ CFLAGS-.o = $(filter %frame-pointer,$(+cflags)) $(pie-default)
++
++# Map Guix store paths to /usr
++CFLAGS-.o += `find /gnu/store -maxdepth 1 -mindepth 1 -type d -exec echo -n " -fdebug-prefix-map={}=/usr" \;`
++
+ libtype.o := lib%.a
+ object-suffixes += .o
+ ifeq (yes,$(build-shared))
+-- 
+2.35.1
+
