Merge pull request #6422

c73a8ea Clarifying offline build process using gbuild --url and noting it is not done automatically. (Midnight Magic)

Author: laanwj

doc/gitian-building.md

@@ -330,10 +330,11 @@ There will be a lot of warnings printed during the build of the image. These can
 Getting and building the inputs
 --------------------------------
 
-Follow the instructions in [doc/release-process.md](release-process.md#fetch-and-build-inputs-first-time-or-when-dependency-versions-change) 
-in the bitcoin repository to install sources which require manual intervention. Also follow
-the next step: 'Seed the Gitian sources cache', which will fetch all the necessary source 
-files to allow gitian to work offline.
+Follow the instructions in [doc/release-process.md](release-process.md#fetch-and-build-inputs-first-time-or-when-dependency-versions-change)
+in the bitcoin repository under 'Fetch and build inputs' to install sources which require
+manual intervention. Also optionally follow the next step: 'Seed the Gitian sources cache
+and offline git repositories' which will fetch the remaining files required for building 
+offline.
 
 Building Bitcoin
 ----------------
@@ -391,6 +392,57 @@ COMMIT=2014_03_windows_unicode_path
 ./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
 ```
 
+Building fully offline
+-----------------------
+
+For building fully offline including attaching signatures to unsigned builds, the detached-sigs repository 
+and the bitcoin git repository with the desired tag must both be available locally, and then gbuild must be
+told where to find them. It also requires an apt-cacher-ng which is fully-populated but set to offline mode, or 
+manually disabling gitian-builder's use of apt-get to update the VM build environment.
+
+To configure apt-cacher-ng as an offline cacher, you will need to first populate its cache with the relevant
+files. You must additionally patch target-bin/bootstrap-fixup to set its apt sources to something other than
+plain archive.ubuntu.com: us.archive.ubuntu.com works.
+
+So, if you use LXC:
+
+```bash
+export PATH="$PATH":/path/to/gitian-builder/libexec
+export USE_LXC=1
+cd /path/to/gitian-builder
+./libexec/make-clean-vm --suite precise --arch amd64
+
+LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root apt-get update
+LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root \
+  -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install \
+  $( sed -ne '/^packages:/,/[^-] .*/ {/^- .*/{s/"//g;s/- //;p}}' ../bitcoin/contrib/gitian-descriptors/*|sort|uniq )
+LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root apt-get -q -y purge grub
+LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade 
+```
+
+And then set offline mode for apt-cacher-ng:
+
+```
+/etc/apt-cacher-ng/acng.conf
+[...]
+Offlinemode: 1
+[...]
+
+service apt-cacher-ng restart
+```
+
+Then when building, override the remote URLs that gbuild would otherwise pull from the gitian descriptors::
+```bash
+
+cd /some/root/path/
+git clone https://github.com/bitcoin/bitcoin-detached-sigs.git
+
+BTCPATH=/some/root/path/bitcoin.git
+SIGPATH=/some/root/path/bitcoin-detached-sigs.git
+
+./bin/gbuild --url bitcoin=${BTCPATH},signature=${SIGPATH} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
+```
+
 Signing externally
 -------------------
 

doc/release-process.md

@@ -6,39 +6,54 @@ Release Process
 
 * * *
 
-###update (commit) version in sources
+###first time only or for new builders, check out the source in the following directory hierarchy
 
+	cd /path/to/your/toplevel/build
+	git clone https://github.com/bitcoin/gitian.sigs.git
+	git clone https://github.com/devrandom/gitian-builder.git
+	git clone https://github.com/bitcoin/bitcoin.git
+
+###for bitcoin maintainers/release engineers, update (commit) version in sources
+
+	pushd ./bitcoin
 	contrib/verifysfbinaries/verify.sh
 	doc/README*
 	share/setup.nsi
 	src/clientversion.h (change CLIENT_VERSION_IS_RELEASE to true)
 
-###tag version in git
+###for bitcoin maintainers/release engineers, tag version in git
 
 	git tag -s v(new version, e.g. 0.8.0)
 
-###write release notes. git shortlog helps a lot, for example:
+###for bitcoin maintainers/release engineers, write release notes. git shortlog helps a lot, for example:
 
 	git shortlog --no-merges v(current version, e.g. 0.7.2)..v(new version, e.g. 0.8.0)
+	popd
 
 * * *
 
-###update gitian
-
- In order to take advantage of the new caching features in gitian, be sure to update to a recent version (`e9741525c` or later is recommended)
+###update gitian, gitian.sigs, checkout bitcoin version, and perform gitian builds
 
-###perform gitian builds
-
- From a directory containing the bitcoin source, gitian-builder and gitian.sigs
+ To ensure your gitian descriptors are accurate for direct reference for gbuild, below, run the following from a directory containing the bitcoin source:
 
+	pushd ./bitcoin
 	export SIGNER=(your gitian key, ie bluematt, sipa, etc)
 	export VERSION=(new version, e.g. 0.8.0)
-	pushd ./bitcoin
 	git checkout v${VERSION}
 	popd
+
+  Ensure your gitian.sigs are up-to-date if you wish to gverify your builds against other gitian signatures:
+
+	pushd ./gitian.sigs
+	git pull
+	popd
+
+  Ensure your gitian-builder sources are up-to-date to take advantage of the new caching features of gitian (`e9741525c` or later is recommended)
+
 	pushd ./gitian-builder
+	git pull
 
-###fetch and build inputs: (first time, or when dependency versions change)
+###fetch and create inputs: (first time, or when dependency versions change)
 
 	mkdir -p inputs
 	wget -P inputs https://bitcoincore.org/cfields/osslsigncode-Backports-to-1.7.1.patch
@@ -52,28 +67,44 @@ Release Process
 
 	tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.9.sdk.tar.gz MacOSX10.9.sdk
 
-###Optional: Seed the Gitian sources cache
+###Optional: Seed the Gitian sources cache and offline git repositories
 
-  By default, gitian will fetch source files as needed. For offline builds, they can be fetched ahead of time:
+By default, gitian will fetch source files as needed. To cache them ahead of time:
 
 	make -C ../bitcoin/depends download SOURCES_PATH=`pwd`/cache/common
 
-  Only missing files will be fetched, so this is safe to re-run for each build.
+Only missing files will be fetched, so this is safe to re-run for each build.
+
+Clone the detached-sigs repository:
+
+	popd
+	git clone https://github.com/bitcoin/bitcoin-detached-sigs.git
+	pushd ./bitcoin-builder
+
+NOTE: Offline builds must use the --url flag to ensure gitian fetches only from local URLs.
+For example: ./bin/bguild --url bitcoin=/path/to/bitcoin,signature=/path/to/sigs {rest of arguments}
+The following gbuild invocations DO NOT DO THIS by default.
 
-###Build Bitcoin Core for Linux, Windows, and OS X:
+###Build (and optionally verify) Bitcoin Core for Linux, Windows, and OS X:
 
 	./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
 	./bin/gsign --signer $SIGNER --release ${VERSION}-linux --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
+	./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-linux ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
 	mv build/out/bitcoin-*.tar.gz build/out/src/bitcoin-*.tar.gz ../
+
 	./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
 	./bin/gsign --signer $SIGNER --release ${VERSION}-win-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
+	./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
 	mv build/out/bitcoin-*-win-unsigned.tar.gz inputs/bitcoin-win-unsigned.tar.gz
 	mv build/out/bitcoin-*.zip build/out/bitcoin-*.exe ../
+
 	./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
 	./bin/gsign --signer $SIGNER --release ${VERSION}-osx-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
+	./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
 	mv build/out/bitcoin-*-osx-unsigned.tar.gz inputs/bitcoin-osx-unsigned.tar.gz
 	mv build/out/bitcoin-*.tar.gz build/out/bitcoin-*.dmg ../
 	popd
+
   Build output expected:
 
   1. source tarball (bitcoin-${VERSION}.tar.gz)
@@ -98,19 +129,21 @@ Commit your signature to gitian.sigs:
 	Once the Windows/OSX builds each have 3 matching signatures, they will be signed with their respective release keys.
 	Detached signatures will then be committed to the bitcoin-detached-sigs repository, which can be combined with the unsigned apps to create signed binaries.
 
-  Create the signed OSX binary:
+  Create (and optionally verify) the signed OSX binary:
 
 	pushd ./gitian-builder
 	./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
 	./bin/gsign --signer $SIGNER --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
+	./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
 	mv build/out/bitcoin-osx-signed.dmg ../bitcoin-${VERSION}-osx.dmg
 	popd
 
-  Create the signed Windows binaries:
+  Create (and optionally verify) the signed Windows binaries:
 
 	pushd ./gitian-builder
 	./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
 	./bin/gsign --signer $SIGNER --release ${VERSION}-win-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
+	./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-signed ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
 	mv build/out/bitcoin-*win64-setup.exe ../bitcoin-${VERSION}-win64-setup.exe
 	mv build/out/bitcoin-*win32-setup.exe ../bitcoin-${VERSION}-win32-setup.exe
 	popd