contrib: Add deterministic Guix builds.

Author: dongcarl

contrib/guix/README.md

@@ -0,0 +1,202 @@
+# Bootstrappable Bitcoin Core Builds
+
+This directory contains the files necessary to perform bootstrappable Bitcoin
+Core builds.
+
+[Bootstrappability][b17e] furthers our binary security guarantees by allowing us
+to _audit and reproduce_ our toolchain instead of blindly _trusting_ binary
+downloads.
+
+We achieve bootstrappability by using Guix as a functional package manager.
+
+## Requirements
+
+Conservatively, a x86_64 machine with:
+
+- 2 or more logical cores
+- 4GB of free disk space on the partition that /gnu/store will reside in
+- 24GB of free disk space on the partition that the bitcoin repository resides in
+
+> Note: these requirements are slightly less onerous than those of Gitian builds
+
+## Setup
+
+**If you're just testing this out, you can use the
+[Dockerfile][fanquake/guix-docker] for convenience. It automatically speeds up
+your builds by [using substitutes](#speeding-up-builds-with-substitute-servers).
+Should you choose to use the Dockerfile, you can skip this section.**
+
+Otherwise, follow the [Guix installation guide][guix/bin-install].
+
+> Note: For those who like to keep their filesystems clean, Guix is designed to
+> be very standalone and _will not_ conflict with your system's package
+> manager/existing setup. It _only_ touches `/var/guix`, `/gnu`, and
+> `~/.config/guix`.
+
+After installation, you may want to consider [adding substitute
+servers](#speeding-up-builds-with-substitute-servers) to speed up your build if
+that fits your security model. (skippable if you're using the
+[Dockerfile][fanquake/guix-docker])
+
+Once Guix is installed, deploy our patched version into your current Guix
+profile. The changes there are slowly being upstreamed.
+
+```sh
+guix pull --url=https://github.com/dongcarl/guix.git \
+          --branch=2019-05-bitcoin-staging \
+          --max-jobs=4 # change accordingly
+```
+
+Make sure that you are using your current profile. (You are prompted to do this
+at the end of the `guix pull`)
+
+```bash
+export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
+```
+
+> Note: There is ongoing work to eliminate this `guix pull` step using Guix
+> [inferiors][guix/inferiors] and [channels][guix/channels].
+
+## Usage
+
+### As a Development Environment
+
+For a Bitcoin Core depends development environment, simply invoke
+
+```sh
+guix environment --manifest=contrib/guix/manifest.scm
+```
+
+And you'll land back in your shell with all the build dependencies required for
+a `depends` build injected into your environment.
+
+### As a Tool for Deterministic Builds
+
+From the top of a clean Bitcoin Core repository:
+
+```sh
+./contrib/guix/guix-build.sh
+```
+
+After the build finishes successfully (check the status code please), compare
+hashes:
+
+```sh
+find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
+```
+
+#### Recognized environment variables
+
+* _**HOSTS**_
+
+  Override the space-separated list of platform triples for which to perform a
+  bootstrappable build. _(defaults to "i686-linux-gnu x86\_64-linux-gnu
+  arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu")_
+
+  > Windows and OS X platform triplet support are WIP.
+
+* _**SOURCES_PATH**_
+
+  Set the depends tree download cache for sources. This is passed through to the
+  depends tree. Setting this to the same directory across multiple builds of the
+  depends tree can eliminate unnecessary redownloading of package sources.
+
+* _**MAX_JOBS**_
+
+  Override the maximum number of jobs to run simultaneously, you might want to
+  do so on a memory-limited machine. This may be passed to `make` as in `make
+  --jobs="$MAX_JOBS"` or `xargs` as in `xargs -P"$MAX_JOBS"`. _(defaults to the
+  value of `nproc` outside the container)_
+
+* _**SOURCE_DATE_EPOCH**_
+
+  Override the reference timestamp used for bit-for-bit reproducibility, the
+  variable name conforms to [standard][r12e/source-date-epoch]. _(defaults to
+  the output of `$(git log --format=%at -1)`)_
+
+* _**V**_
+
+  If non-empty, will pass `V=1` to all `make` invocations, making `make` output
+  verbose.
+
+* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_
+
+  Additional flags to be passed to `guix environment`. For a fully-bootstrapped
+  build, set this to `--bootstrap --no-substitutes`. Note that a
+  fully-bootstrapped build will take quite a long time on the first run.
+
+## Tips and Tricks
+
+### Speeding up builds with substitute servers
+
+_This whole section is automatically done in the convenience
+[Dockerfiles][fanquake/guix-docker]_
+
+For those who are used to life in the fast _(and trustful)_ lane, you can use
+[substitute servers][guix/substitutes] to enable binary downloads of packages.
+
+> For those who only want to use substitutes from the official Guix build farm
+> and have authorized the build farm's signing key during Guix's installation,
+> you don't need to do anything.
+
+#### Authorize the signing keys
+
+For the official Guix build farm at https://ci.guix.gnu.org, run as root:
+
+```
+guix archive --authorize < ~root/.config/guix/current/share/guix/ci.guix.gnu.org.pub
+```
+
+For dongcarl's substitute server at https://guix.carldong.io, run as root:
+
+```sh
+wget -qO- 'https://guix.carldong.io/signing-key.pub' | guix archive --authorize
+```
+
+#### Use the substitute servers
+
+The official Guix build farm at https://ci.guix.gnu.org is automatically used
+unless the `--no-substitutes` flag is supplied.
+
+This can be overridden for all `guix` invocations by passing the
+`--substitute-urls` option to your invocation of `guix-daemon`. This can also be
+overridden on a call-by-call basis by passing the same `--substitute-urls`
+option to client tools such at `guix environment`.
+
+To use dongcarl's substitute server for Bitcoin Core builds after having
+[authorized his signing key](#authorize-the-signing-keys):
+
+```
+export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--substitute-urls="https://guix.carldong.io https://ci.guix.gnu.org"'
+```
+
+## FAQ
+
+### How can I trust the binary installation?
+
+As mentioned at the bottom of [this manual page][guix/bin-install]:
+
+> The binary installation tarballs can be (re)produced and verified simply by
+> running the following command in the Guix source tree:
+>
+>     make guix-binary.x86_64-linux.tar.xz
+
+### When will Guix be packaged in debian?
+
+Vagrant Cascadian has been making good progress on this
+[here][debian/guix-package]. We have all the pieces needed to put up an APT
+repository and will likely put one up soon.
+
+[b17e]: http://bootstrappable.org/
+[r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/
+
+[guix/install.sh]: https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
+[guix/bin-install]: https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html
+[guix/env-setup]: https://www.gnu.org/software/guix/manual/en/html_node/Build-Environment-Setup.html
+[guix/substitutes]: https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html
+[guix/substitute-server-auth]: https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html
+[guix/inferiors]: https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html
+[guix/channels]: https://www.gnu.org/software/guix/manual/en/html_node/Channels.html
+
+[debian/guix-package]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850644
+[fanquake/guix-docker]: https://github.com/fanquake/core-review/tree/master/guix

contrib/guix/guix-build.sh

@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+export LC_ALL=C
+set -e -o pipefail
+
+# Determine the maximum number of jobs to run simultaneously (overridable by
+# environment)
+MAX_JOBS="${MAX_JOBS:-$(nproc)}"
+
+# Download the depends sources now as we won't have internet access in the build
+# container
+make -C "${PWD}/depends" -j"$MAX_JOBS" download ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
+
+# Determine the reference time used for determinism (overridable by environment)
+SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}"
+
+# Deterministically build Bitcoin Core for HOSTs (overriable by environment)
+for host in ${HOSTS=i686-linux-gnu x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu}; do
+
+    # Display proper warning when the user interrupts the build
+    trap 'echo "** INT received while building ${host}, you may want to clean up the relevant output and distsrc-* directories before rebuilding"' INT
+
+    # Run the build script 'contrib/guix/build.sh' in the build container
+    # specified by 'contrib/guix/manifest.scm'
+    # shellcheck disable=SC2086
+    guix environment --manifest="${PWD}/contrib/guix/manifest.scm" \
+                     --container \
+                     --pure \
+                     --no-cwd \
+                     --share="$PWD"=/bitcoin \
+                     ${SOURCES_PATH:+--share="$SOURCES_PATH"} \
+                     ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
+                     -- env HOST="$host" \
+                            MAX_JOBS="$MAX_JOBS" \
+                            SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \
+                            ${V:+V=1} \
+                            ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \
+                          bash -c "cd /bitcoin && bash contrib/guix/libexec/build.sh"
+
+done