Skip to content

Commit c7597d2

Browse files
quakequake
authored
Merge pull request #11 from cryptape/reproducible-build-via-docker
Add a sample reproducible build solution using docker
2 parents 1394be8 + cc85ede commit c7597d2

File tree

6 files changed

+208
-2
lines changed

6 files changed

+208
-2
lines changed

‎.github/workflows/rust.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,14 @@ jobs:
3737
git submodule add https://github.com/xxuejie/lib-dummy-atomics deps/lib-dummy-atomics
3838
- name: Run all checks
3939
run: cd test-workspace && make build test check clippy
40+
- name: Reproducible build runs
41+
run: cd test-workspace && ./scripts/reproducible_build_docker --update && ./scripts/reproducible_build_docker --no-clean
4042
- name: Generate standalone contract
4143
run: cargo generate --path . standalone-contract --name test-contract
4244
- name: Run all checks
4345
run: cd test-contract && make build test check clippy
46+
- name: Reproducible build runs
47+
run: cd test-contract && ./scripts/reproducible_build_docker --update && ./scripts/reproducible_build_docker --no-clean
4448

4549
debian-build:
4650

@@ -198,6 +202,49 @@ jobs:
198202
- name: Run all checks
199203
run: cd test-contract && make build test check clippy
200204

205+
# For now, github action's latest arm64-based macos runner does not support docker. We will have to
206+
# Use an order version for now to test reproducible build on macOS. Maybe when macOS 15 is out we can
207+
# re-test later.
208+
#
209+
# References:
210+
# * https://github.com/orgs/community/discussions/69211
211+
# * https://github.com/marketplace/actions/setup-docker-on-macos
212+
macos-x64-reproducible-build:
213+
214+
runs-on: macos-13
215+
216+
steps:
217+
- uses: actions/checkout@v3
218+
with:
219+
submodules: true
220+
- name: Setup Docker on macOS
221+
run: brew install docker colima && colima start
222+
- name: Install llvm 16
223+
run: brew install llvm@16
224+
- name: Install riscv64 target
225+
run: rustup target add riscv64imac-unknown-none-elf
226+
- name: Install cargo generate
227+
run: cargo install cargo-generate
228+
- name: Generate workspace
229+
run: cargo generate --path . workspace --name test-workspace
230+
- name: Generate crates && contracts
231+
run: cd test-workspace &&
232+
make generate CRATE=clib TEMPLATE=c-wrapper-crate DESTINATION=crates TEMPLATE_TYPE=--path TEMPLATE_REPO=.. &&
233+
make generate CRATE=rlib TEMPLATE=x64-simulator-crate DESTINATION=crates TEMPLATE_TYPE=--path TEMPLATE_REPO=.. &&
234+
make generate CRATE=c1 TEMPLATE=contract TEMPLATE_TYPE=--path TEMPLATE_REPO=.. &&
235+
make generate CRATE=c2 TEMPLATE=atomics-contract TEMPLATE_TYPE=--path TEMPLATE_REPO=.. &&
236+
make generate CRATE=c3 TEMPLATE=stack-reorder-contract TEMPLATE_TYPE=--path TEMPLATE_REPO=..
237+
- name: Submodules
238+
run: cd test-workspace &&
239+
git submodule add https://github.com/nervosnetwork/ckb-c-stdlib deps/ckb-c-stdlib &&
240+
git submodule add https://github.com/xxuejie/lib-dummy-atomics deps/lib-dummy-atomics
241+
- name: Reproducible build runs
242+
run: cd test-workspace && ./scripts/reproducible_build_docker --update && ./scripts/reproducible_build_docker --no-clean
243+
- name: Generate standalone contract
244+
run: cargo generate --path . standalone-contract --name test-contract
245+
- name: Reproducible build runs
246+
run: cd test-contract && ./scripts/reproducible_build_docker --update && ./scripts/reproducible_build_docker --no-clean
247+
201248
windows-build:
202249

203250
runs-on: windows-2019

‎README.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,30 @@ make test
171171

172172
The templates provided here, use the same conventions as `ckb-native-build-sample` project, so feel free to refer to the more detailed [usage](https://github.com/xxuejie/ckb-native-build-sample?tab=readme-ov-file#usage) doc in the sample project.
173173

174+
### Reproducible Build
175+
176+
When using this set of templates, we always recommend to use locally installed native versions of LLVM & Rust to build and test your scripts. However, reproducible build is an important part of CKB scripts, which would require locked versions of LLVM & Rust to work. It might not be an easy task when using locally installed versions of compilers.
177+
178+
For the time being, we have prepared a script that does reproducible build via [a docker container image](https://github.com/cryptape/llvm-n-rust). We do want to mention that docker is not necessarily THE way to do reproducible build, nor is it the best way to do reproducible build. There might well be other ways that are better, such as chroot or Nix. It's just that historically, docker has been used in CKB script's build process, and adding a script leveraging docker here, provides an easy solution into the issue.
179+
180+
To do reproducible build, you can use the included script with varying commands:
181+
182+
```
183+
$ ./scripts/reproducible_build_docker # Clean current repository, used locked LLVM & Rust from a docker container
184+
# to build all contracts, then test the binaries against a checksum file.
185+
186+
$ ./scripts/reproducible_build_docker --update # Update the checksum file with new binaries, could be handy when you have
187+
# made changes to the binaries.
188+
189+
$ ./scripts/reproducible_build_docker --no-clean # Do not clean intermediate files before building, it is not recommended to
190+
# use this but when you really know what you are doing, it could help you save
191+
# some time.
192+
193+
$ ./scripts/reproducible_build_docker --proxy "..." # Setup docker container so it pulls Rust crates using a proxy server
194+
```
195+
196+
By default, the checksum file is stored in `checksums.txt` in the root of the repository. It is strongly recommended that this file is checked into version control, and a CI is setup so reproducible build is always checked in new commits.
197+
174198
### Standalone Contract Crate
175199

176200
In rare cases if you want to simply use a standalone contract crate without a workspace. The [standalone-contract](https://github.com/cryptape/ckb-script-templates/tree/main/standalone-contract) template is prepared for you:

‎standalone-contract/Makefile‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ BUILD_DIR := build/$(MODE)
2424
# likely match the crate name, which is also the name of the final binary.
2525
# However if this is not the case, you can tweak this variable. As the name hints,
2626
# more than one binary is supported here.
27-
BINARIES := $(notdir $(shell pwd))
27+
BINARIES := {{project-name}}
2828

2929
ifeq (release,$(MODE))
3030
MODE_ARGS := --release
@@ -75,4 +75,9 @@ clean:
7575
prepare:
7676
rustup target add riscv64imac-unknown-none-elf
7777

78+
# Generate checksum info for reproducible build
79+
CHECKSUM_FILE := build/checksums-$(MODE).txt
80+
checksum: build
81+
shasum -a 256 build/$(MODE)/* > $(CHECKSUM_FILE)
82+
7883
.PHONY: build test check clippy fmt cargo clean prepare

‎standalone-contract/scripts/reproducible_build_docker‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
#!/usr/bin/env bash
2+
#
3+
# An utility script helping with reproducible script builds via docker.
4+
# Note that this utility serves only as one example, docker is not
5+
# necessarily THE way to do reproducible build, nor is it the best way
6+
# to do reproducible build.
7+
set -ex
8+
9+
DOCKER="${DOCKER:-docker}"
10+
DOCKER_IMAGE="${DOCKER_IMAGE:-docker.io/cryptape/llvm-n-rust:20240630}"
11+
CHECKSUM_FILE_PATH="${CHECKSUM_FILE_PATH:-checksums.txt}"
12+
13+
# We are parsing command line arguments based on tips from:
14+
# https://stackoverflow.com/a/14203146
15+
16+
while [[ $# -gt 0 ]]; do
17+
case $1 in
18+
-p|--proxy)
19+
PROXY="$2"
20+
shift # past argument
21+
shift # past value
22+
;;
23+
-u|--update)
24+
UPDATE="yes"
25+
shift # past argument
26+
;;
27+
--no-clean)
28+
NOCLEAN="yes"
29+
shift # past argument
30+
;;
31+
-*|--*)
32+
echo "Unknown option $1"
33+
exit 1
34+
;;
35+
*)
36+
echo "Unknown argument $1"
37+
exit 1
38+
;;
39+
esac
40+
done
41+
42+
if [[ -n "${PROXY}" ]]; then
43+
DOCKER_RUN_ARGS="-e ALL_PROXY=${PROXY} -e HTTPS_PROXY=${PROXY} -e HTTP_PROXY=${PROXY} ${DOCKER_RUN_ARGS}"
44+
fi
45+
46+
TASKS=""
47+
if [[ "${NOCLEAN}" != "yes" ]]; then
48+
TASKS+=" clean "
49+
fi
50+
51+
if [[ "${UPDATE}" = "yes" ]]; then
52+
TASKS+=" checksum CHECKSUM_FILE=${CHECKSUM_FILE_PATH} "
53+
else
54+
TASKS+=" build "
55+
fi
56+
57+
$DOCKER run --rm $DOCKER_RUN_ARGS -v `pwd`:/code $DOCKER_IMAGE make $TASKS
58+
# Reset file ownerships for all files docker might touch
59+
$DOCKER run --rm $DOCKER_RUN_ARGS -e UID=`id -u` -e GID=`id -g` -v `pwd`:/code $DOCKER_IMAGE bash -c 'chown -R -f $UID:$GID checksums.txt build target'
60+
61+
if [[ "${UPDATE}" = "yes" ]]; then
62+
echo "${CHECKSUM_FILE_PATH} file is updated with latest binary hashes!"
63+
else
64+
shasum -a 256 -c ${CHECKSUM_FILE_PATH}
65+
fi

‎workspace/Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,6 @@ prepare:
110110
# Generate checksum info for reproducible build
111111
CHECKSUM_FILE := build/checksums-$(MODE).txt
112112
checksum: build
113-
sha256sum build/$(MODE)/* > $(CHECKSUM_FILE)
113+
shasum -a 256 build/$(MODE)/* > $(CHECKSUM_FILE)
114114

115115
.PHONY: build test check clippy fmt cargo clean prepare checksum

‎workspace/scripts/reproducible_build_docker‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
#!/usr/bin/env bash
2+
#
3+
# An utility script helping with reproducible script builds via docker.
4+
# Note that this utility serves only as one example, docker is not
5+
# necessarily THE way to do reproducible build, nor is it the best way
6+
# to do reproducible build.
7+
set -ex
8+
9+
DOCKER="${DOCKER:-docker}"
10+
DOCKER_IMAGE="${DOCKER_IMAGE:-docker.io/cryptape/llvm-n-rust:20240630}"
11+
CHECKSUM_FILE_PATH="${CHECKSUM_FILE_PATH:-checksums.txt}"
12+
13+
# We are parsing command line arguments based on tips from:
14+
# https://stackoverflow.com/a/14203146
15+
16+
while [[ $# -gt 0 ]]; do
17+
case $1 in
18+
-p|--proxy)
19+
PROXY="$2"
20+
shift # past argument
21+
shift # past value
22+
;;
23+
-u|--update)
24+
UPDATE="yes"
25+
shift # past argument
26+
;;
27+
--no-clean)
28+
NOCLEAN="yes"
29+
shift # past argument
30+
;;
31+
-*|--*)
32+
echo "Unknown option $1"
33+
exit 1
34+
;;
35+
*)
36+
echo "Unknown argument $1"
37+
exit 1
38+
;;
39+
esac
40+
done
41+
42+
if [[ -n "${PROXY}" ]]; then
43+
DOCKER_RUN_ARGS="-e ALL_PROXY=${PROXY} -e HTTPS_PROXY=${PROXY} -e HTTP_PROXY=${PROXY} ${DOCKER_RUN_ARGS}"
44+
fi
45+
46+
TASKS=""
47+
if [[ "${NOCLEAN}" != "yes" ]]; then
48+
TASKS+=" clean "
49+
fi
50+
51+
if [[ "${UPDATE}" = "yes" ]]; then
52+
TASKS+=" checksum CHECKSUM_FILE=${CHECKSUM_FILE_PATH} "
53+
else
54+
TASKS+=" build "
55+
fi
56+
57+
$DOCKER run --rm $DOCKER_RUN_ARGS -v `pwd`:/code $DOCKER_IMAGE make $TASKS
58+
# Reset file ownerships for all files docker might touch
59+
$DOCKER run --rm $DOCKER_RUN_ARGS -e UID=`id -u` -e GID=`id -g` -v `pwd`:/code $DOCKER_IMAGE bash -c 'chown -R -f $UID:$GID checksums.txt build target'
60+
61+
if [[ "${UPDATE}" = "yes" ]]; then
62+
echo "${CHECKSUM_FILE_PATH} file is updated with latest binary hashes!"
63+
else
64+
shasum -a 256 -c ${CHECKSUM_FILE_PATH}
65+
fi

0 commit comments

Comments
 (0)