Add job checking for reproducibility issues#2342
Merged
emilyalbini merged 1 commit intomasterfrom Jan 9, 2026
Merged
Conversation
emilyalbini
force-pushed
the
ea-repro
branch
20 times, most recently
from
f2b3afd to
e452eda
Compare
emilyalbini
changed the title
labbott
reviewed
Jan 8, 2026
.github/workflows/ci.yml
Outdated
Comment on lines
123
to
125
| # Hubris builds have to be reproducible, and we want to test that in CI. This job does a build of | ||
| # an arbitrary board (specifically cosmo-b, but it can be changed to any board) with the standard | ||
| # Ubuntu image and no interference, as the baseline to compire to. |
Collaborator
There was a problem hiding this comment.
Any chance we could also test an RoT board? There's enough disjoint code between SP and RoT and I could unfortunately see us accidentally sneaking in something non-reproducible to the RoT. Arguably there is some amount of disjoint code between e.g. cosmo and sidecar but that amount is smaller than SP vs RoT.
Member
Author
There was a problem hiding this comment.
Done! Added both the RoT and Cosmo. It should also be trivial to add more in the future by just adding extra boards to the matrix.
| # | ||
| # We also use disorderfs to randomize the ordering of listing directories, to catch code | ||
| # assuming directory entries are always returned in the same order. | ||
| - name: Prepare a custom build root directory with disorderfs |
Collaborator
There was a problem hiding this comment.
TIL disorderfs. This is lightly terrifying and also useful!
emilyalbini
force-pushed
the
ea-repro
branch
4 times, most recently
from
4bd8a5f to
0d2dfec
Compare
labbott
approved these changes
Jan 9, 2026
Collaborator
labbott
left a comment
labbott
left a comment
There was a problem hiding this comment.
LGTM let's give this a shot 😎
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a set of CI jobs to try and prevent nondeterminism in the build process. While they are not a guarantee that Hubris will be reproducible, it will hopefully catch most cases where we accidentally introduce nondeterminism.
There are two new jobs, reproducible-a and reproducible-b, building the same board (I arbitrarily picked cosmo-b, it can be anything). The first job uses the standard Ubuntu runner with little to no modification, while the second job tries to mutate the build environment as much as possible to introduce sources of nondeterminism. The code of the second job is heavily commented to describe each source and how we introduce it.
Inspiration for which variations to introduce was inspired by both past personal experience and the documentation provided by the reproducible-builds project. There is a source I wanted to include (different UIDs) but didn't, and that's because running things as different users inside of GitHub Actions was breaking too many things to be worth it.
There is then a final new job, reproducible-check, that gets the artifacts from both builders and compares them with diffoscope, another tool by the reproducible-builds project. In case there are differences it will upload the diffoscope report as an artifact for analysis. This is a run where I intentionally induced nondeterminism, if you want to look at how things would be in case of a failure (there is a link to the diffoscope report in there).
If there are other sources of nondeterminism we want to try and inject let me know and I'll see how feasible it is to add them.
Fixes #2299