Skip to content

Commit f2b3afd

Browse files
emilyalbiniemilyalbini
committed
add job checking for reproducibility issues
1 parent b209b32 commit f2b3afd

File tree

1 file changed

+162
-0
lines changed

1 file changed

+162
-0
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,167 @@ jobs:
102102
uses: actions/deploy-pages@v4
103103
id: deployment
104104

105+
# Hubris builds have to be reproducible, and we want to test that in CI. This job does a build of
106+
# an arbitrary board (specifically cosmo-b, but it can be changed to any board) with the standard
107+
# Ubuntu image and no interference, as the baseline to compire to.
108+
reproducible-a:
109+
name: Reproducibility (A)
110+
runs-on: ubuntu-latest
111+
permissions:
112+
contents: read
113+
steps:
114+
- name: Checkout the source code
115+
uses: actions/checkout@v6
116+
117+
# We check explicitly to ensure the other job has a different one.
118+
- name: Check that GCC is the system C toolchain
119+
run: cc --version | grep -q "Free Software Foundation"
120+
121+
- name: Build a Hubris board
122+
run: |
123+
umask 0007 # We set the umask explicitly here to ensure the other job has a different one.
124+
cargo xtask dist app/cosmo/rev-b.toml
125+
126+
- name: Upload the artifact to be later checked
127+
uses: actions/upload-artifact@v6
128+
with:
129+
name: reproducible-a
130+
path: target/cosmo-b/dist/default/build-cosmo-b-image-default.zip
131+
if-no-files-found: error
132+
133+
# Hubris builds have to be reproducible, and we want to test that in CI. This job does a build of
134+
# an arbitrary board (specifically cosmo-b, but it can be changed to any board) trying to change
135+
# the build environment as further as possible from reproducible-a. Each variability we introduce
136+
# and its reasoning is documented in comments below.
137+
#
138+
# More information on common variations are available on the reproducible-builds website:
139+
# https://reproducible-builds.org/docs/env-variations/
140+
reproducible-b:
141+
name: Reproducibility (B)
142+
runs-on: ubuntu-latest
143+
permissions:
144+
contents: read
145+
env:
146+
CUSTOM_ROOT: /very/long/path/we/are/doing/the/build/in/to/check/for/issues/with/long/paths/or/different/paths
147+
steps:
148+
- name: Install Ubuntu dependencies
149+
run: |
150+
sudo apt-get update
151+
sudo apt-get install -y disorderfs clang
152+
sudo apt-get remove -y gcc
153+
sudo apt-get autoremove -y
154+
155+
# In the Ubuntu dependencies installation step above we switched from GCC to Clang as the
156+
# system C toolchain and linker. We are not using the system linker in the build process
157+
# (Hubris uses the LLD copy bundled with Rust), so switching the system toolchain will catch
158+
# us accidentally relying on it (and breaking reproducibility depending on which toolchain is
159+
# installed on the system attempting to reproduce).
160+
- name: Check that clang is the system C toolchain
161+
run: |
162+
! command -v gcc >/dev/null
163+
cc --version | grep -q clang
164+
165+
# GitHub Actions is really annoying and doesn't let us clone repositories with the checkout
166+
# action outside of $GITHUB_WORKSPACE, erroring out if we dare to do so. We still want to use
167+
# the action though, as it will gracefully handle authentication if someone creates a private
168+
# clone of Hubris. We thus clone the repository in the default path, and then in a later step
169+
# clone this checkout into the custom build directory.
170+
- name: Checkout the source code in the standard GitHub Actions directory
171+
uses: actions/checkout@v6
172+
173+
# We run the Hubris build in a different directory, to ensure that paths are not hardcoded. We
174+
# also use a very long path, as a reproducibility issue Emily found in the wild in the past
175+
# was a rust-lang/rust test failing when built in a path that was too long.
176+
#
177+
# We also use disorderfs to randomize the ordering of listing directories, to catch code
178+
# assuming directory entries are always returned in the same order.
179+
- name: Prepare a custom build root directory with disorderfs
180+
run: |
181+
sudo mkdir -p $CUSTOM_ROOT
182+
sudo disorderfs --multi-user=yes --shuffle-dirents=yes $(pwd) $CUSTOM_ROOT
183+
184+
# The current time might be included in the built artifacts. To ensure reproducibility, move
185+
# the time forward by a day and a few hours. This should be enough to expose differences in
186+
# the build without messing with TLS certificate expiration.
187+
#
188+
# Note that this causes very funny behavior in GitHub Action's workflow UI, as apparently step
189+
# duration estimates are based on time reported by the runner???
190+
- name: Move forward in time to ensure a different build date
191+
run: |
192+
sudo timedatectl set-ntp false
193+
sudo timedatectl set-time "$(date -d '1 day ago 11 hours ago' "+%Y-%m-%d %H:%M:%S")"
194+
date
195+
196+
- name: Build a Hubris board
197+
run: |
198+
# Permissions of files created during the build process might leak into the artifacts.
199+
# Changing the umask will let us test with different permissions than archives created in
200+
# the reproducible-a job.
201+
umask 0077
202+
203+
cd $CUSTOM_ROOT
204+
cargo xtask dist app/cosmo/rev-b.toml
205+
206+
- name: Move back to the right time
207+
run: sudo timedatectl set-ntp true
208+
209+
- name: Upload the artifact to be later checked
210+
uses: actions/upload-artifact@v6
211+
with:
212+
name: reproducible-b
213+
path: ${{ env.CUSTOM_ROOT }}/target/cosmo-b/dist/default/build-cosmo-b-image-default.zip
214+
if-no-files-found: error
215+
216+
reproducible-check:
217+
name: Reproducibility check
218+
runs-on: ubuntu-slim
219+
needs:
220+
- reproducible-a
221+
- reproducible-b
222+
permissions: {}
223+
steps:
224+
- name: Install uv (Python package manager)
225+
uses: astral-sh/setup-uv@v7
226+
with:
227+
enable-cache: false
228+
ignore-empty-workdir: true
229+
230+
- name: Download reproducible artifacts
231+
uses: actions/download-artifact@v7
232+
with:
233+
pattern: reproducible-*
234+
235+
# Diffoscope is a tool built by the reproducible-builds people to do a rich format-aware diff
236+
# of two files. For example, it understands both zip archives and ELF objects, so it can point
237+
# out the member of archive or the ELF section containing the difference.
238+
#
239+
# We are pulling diffoscope from PyPI transparently through `uvx`, instead of installing it
240+
# from the Ubuntu archives with `apt-get`. We're doing this because due to (mostly sensible)
241+
# packaging choice installing the Ubuntu package takes 6+ minutes, compared to the sub-second
242+
# installation time `uvx` provides. The fact we get a newer version doesn't hurt either.
243+
#
244+
# If you are curious, the reason why the Ubuntu package takes so long to install is because
245+
# diffoscope can produce better diffs the more CLI tools are installed, and the Ubuntu package
246+
# depends on all of those tools. We don't really care about all of them, and the barebones
247+
# version installed through PyPI is enough for us.
248+
- name: Compare the two reproducible artifacts
249+
run: uvx diffoscope --html report.html reproducible-a/build-cosmo-b-image-default.zip reproducible-b/build-cosmo-b-image-default.zip
250+
251+
- name: Upload the diffoscope report
252+
if: failure() # Only upload the report if the previous step failed.
253+
id: diffoscope-report
254+
uses: actions/upload-artifact@v6
255+
with:
256+
name: reproducible-diffoscope-report
257+
path: report.html
258+
if-no-files-found: error
259+
260+
- name: Add a job summary to point folks to diffoscope
261+
if: failure()
262+
run: echo "Non-reproducibility was detected by CI. [Download the diffoscope report]($REPORT_URL) to learn more" >> $GITHUB_STEP_SUMMARY
263+
env:
264+
REPORT_URL: ${{ steps.diffoscope-report.outputs.artifact-url }}
265+
105266
finish:
106267
name: CI finished
107268
runs-on: ubuntu-slim
@@ -112,6 +273,7 @@ jobs:
112273
- format
113274
- docs-build
114275
- docs-deploy
276+
- reproducible-check
115277
if: "${{ !cancelled() }}"
116278
steps:
117279
- name: Calculate the correct exit status

0 commit comments

Comments
 (0)