Skip to content

Commit 078d6ea

Browse files
PastaPastaPastaPastaPastaPasta
committed
Merge #6400: ci: refactor GitHub Actions CI to use reusable workflows, respect DEP_OPTS, build multiprocess depends with Clang
a976777 ci: build multiprocess depends with Clang, use in tsan builds (Kittywhiskers Van Gogh) c9708b7 ci: drop `DEBUG=1` to try and avoid exhausting Actions runner storage (Kittywhiskers Van Gogh) 68ddcdb ci: add nowallet depends, multiprocess depends and build variant (Kittywhiskers Van Gogh) c83efa8 ci: start respecting `DEP_OPTS`, unify with GitLab (Kittywhiskers Van Gogh) f41efa4 ci: source target triple from `matrix.sh` (Kittywhiskers Van Gogh) a7d11f8 ci: pass the whole key instead of trying to reconstruct it (Kittywhiskers Van Gogh) fa0dc5e ci: switch from matrices to manual definition (Kittywhiskers Van Gogh) 1f58365 ci: split out building src and running unit tests into reusable workflow (Kittywhiskers Van Gogh) 0d8f2ef ci: split out building depends into reusable workflow (Kittywhiskers Van Gogh) 41d9867 ci: split out building CI container into reusable workflow (Kittywhiskers Van Gogh) Pull request description: ## Additional Information * Depends on #6406 * We have to move away from matrices as matrix-based job variants cannot be _individually_ accepted as dependencies for the next job, only the whole job can. This means that the next job cannot start until the last matrix variant of the dependency succeeds. This also means that if any variant fails in the dependency job fails, every job forward also ceases. The approach taken in this PR is a mirror of the current implementation on GitLab CI, defining a reusable unit ("templates" in GitLab parlance, "reusable workflows" in GitHub) and defining each variant, now coming with a unique job name that can be specified as dependencies for the job. This should reduce stalling from slower variants. * `DEBUG=1` had to be dropped as it resulted in the runners running out of free space ([build](https://github.com/kwvg/dash/actions/runs/13240781702/job/36956549082#step:7:3137)). Despite this only being a problem with GitHub Actions, this change also affects GitLab CI as we now read `DEP_OPTS` from matrix scripts instead of the CI configuration file. * We are reading from the matrix script to ensure that both CI instances build with the same configuration and to reduce code complexity arising from passing around details between different workflows. * Potential `DEP_OPTS` conflicts (as discussed [here](#6400 (comment))) have been circumvented by passing the expected cache key wholesale instead of reconstructing it ([commit](1a67a02)). We do still expect that `HOST` will be defined correctly in the matrix script (the `HOST` of both depends and build should match) but as differing `HOST`s are incompatible regardless, this shouldn't be a problem. * To make sure we can `output` `cache-primary-key`, the depends cache step was split in two so that the output from the `restore` action is can be exported from the workflow ([source](https://github.com/dashpay/dash/blob/1a67a026de39017a544cfb7852390de88864deec/.github/workflows/build-depends.yml#L21)). * As `DEP_OPTS` is a new parameter that _could_ require rebuilding depends, it needs to be incorporated into the cache key. The most straightforward way to do it is to append the hash of the file that defines `DEP_OPTS` to the cache key. Unfortunately, this requires us to hardcode the name of the individual file (e.g., `00_setup_env_native_qt5.sh` for `linux64`) as `matrix.sh` is just a dispatcher and comes with the drawback that _any_ change to the script could result in a cache miss due to a changed hash. This has been mitigated by hashing the build variant name and the environment variables that influence the build ([source](https://github.com/dashpay/dash/blob/cfdd2c678819de04cd867740a0f9a9f729c891bb/.github/workflows/build-depends.yml#L41-L43)). * `head` trims the output of `sha256sum` to 64 characters, the hash itself isn't being trimmed but everything after it is (trailing hyphen and newline). This is also why `echo -n` is used in some places, to avoid newline addition resulting in a different hash value. * Currently there doesn't seem to be a way to have full control on a reusable workflow's name, it remains a fixed "[caller name] / [reusable workflow name]" and attempting to remove the name altogether will still result in a trailing slash ([source](https://stackoverflow.com/questions/79241079/how-to-control-the-name-of-a-reusable-workflow)). ## Breaking Changes None expected. ## Checklist - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)** - [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)** - [x] I have made corresponding changes to the documentation **(note: N/A)** - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_ ACKs for top commit: UdjinM6: ACK a976777 PastaPastaPasta: utACK a976777 Tree-SHA512: 34436d7e887bc12b89cced49c17ec83405c074dd01eddce8a425c4edc67e7fc6005d1ddc36eef0c8c4888d65a5e6c99ae4bce1726ebfa07d8b7a07063fed4f40
2 parents 162ca94 + a976777 commit 078d6ea

File tree

8 files changed

+376
-202
lines changed

8 files changed

+376
-202
lines changed

‎.github/workflows/build-container.yml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
name: Build container
2+
3+
on:
4+
workflow_call:
5+
outputs:
6+
path:
7+
description: "Path to built container"
8+
value: ghcr.io/${{ jobs.build.outputs.repo }}/dashcore-ci-runner:${{ jobs.build.outputs.tag }}
9+
10+
env:
11+
DOCKER_DRIVER: overlay2
12+
13+
jobs:
14+
build:
15+
name: Build container
16+
runs-on: ubuntu-22.04
17+
outputs:
18+
tag: ${{ steps.prepare.outputs.tag }}
19+
repo: ${{ steps.prepare.outputs.repo }}
20+
steps:
21+
- name: Checkout code
22+
uses: actions/checkout@v4
23+
with:
24+
ref: ${{ github.event.pull_request.head.sha }}
25+
26+
- name: Prepare variables
27+
id: prepare
28+
run: |
29+
BRANCH_NAME=$(echo "${GITHUB_REF##*/}" | tr '[:upper:]' '[:lower:]')
30+
REPO_NAME=$(echo "${{ github.repository }}" | tr '[:upper:]' '[:lower:]')
31+
echo "tag=${BRANCH_NAME}" >> $GITHUB_OUTPUT
32+
echo "repo=${REPO_NAME}" >> $GITHUB_OUTPUT
33+
34+
- name: Set up Docker Buildx
35+
uses: docker/setup-buildx-action@v3
36+
37+
- name: Login to GitHub Container Registry
38+
uses: docker/login-action@v3
39+
with:
40+
registry: ghcr.io
41+
username: ${{ github.actor }}
42+
password: ${{ secrets.GITHUB_TOKEN }}
43+
44+
- name: Build and push Docker image
45+
uses: docker/build-push-action@v6
46+
with:
47+
context: ./contrib/containers/ci
48+
file: ./contrib/containers/ci/Dockerfile
49+
push: true
50+
tags: |
51+
ghcr.io/${{ steps.prepare.outputs.repo }}/dashcore-ci-runner:${{ steps.prepare.outputs.tag }}
52+
ghcr.io/${{ steps.prepare.outputs.repo }}/dashcore-ci-runner:latest
53+
cache-from: type=registry,ref=ghcr.io/${{ steps.prepare.outputs.repo }}/dashcore-ci-runner:latest
54+
cache-to: type=inline

‎.github/workflows/build-depends.yml‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
name: Build depends
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
build-target:
7+
description: "Target name as defined by matrix.sh"
8+
required: true
9+
type: string
10+
container-path:
11+
description: "Path to built container at registry"
12+
required: true
13+
type: string
14+
outputs:
15+
key:
16+
description: "Key needed for restoring depends cache"
17+
value: ${{ jobs.build-depends.outputs.key }}
18+
19+
jobs:
20+
build-depends:
21+
name: Build depends
22+
runs-on: ubuntu-22.04
23+
outputs:
24+
key: ${{ steps.restore.outputs.cache-primary-key }}
25+
container:
26+
image: ${{ inputs.container-path }}
27+
options: --user root
28+
steps:
29+
- name: Checkout code
30+
uses: actions/checkout@v4
31+
with:
32+
ref: ${{ github.event.pull_request.head.sha }}
33+
34+
- name: Initial setup
35+
id: setup
36+
run: |
37+
BUILD_TARGET="${{ inputs.build-target }}"
38+
source ./ci/dash/matrix.sh
39+
echo "DEP_OPTS=${DEP_OPTS}" >> "${GITHUB_OUTPUT}"
40+
echo "HOST=${HOST}" >> "${GITHUB_OUTPUT}"
41+
DEP_HASH="$(echo -n "${BUILD_TARGET}" "${DEP_OPTS}" "${HOST}" | sha256sum | head -c 64)"
42+
echo "\"${BUILD_TARGET}\" has HOST=\"${HOST}\" and DEP_OPTS=\"${DEP_OPTS}\" with hash \"${DEP_HASH}\""
43+
echo "DEP_HASH=${DEP_HASH}" >> "${GITHUB_OUTPUT}"
44+
45+
shell: bash
46+
47+
- name: Cache depends sources
48+
uses: actions/cache@v4
49+
with:
50+
path: |
51+
depends/sources
52+
key: depends-sources-${{ hashFiles('depends/packages/*') }}
53+
restore-keys: |
54+
depends-sources-${{ hashFiles('depends/packages/*') }}
55+
depends-sources-
56+
57+
- name: Restore cached depends
58+
uses: actions/cache/restore@v4
59+
id: restore
60+
with:
61+
path: |
62+
depends/built
63+
depends/${{ steps.setup.outputs.HOST }}
64+
key: ${{ runner.os }}-depends-${{ inputs.build-target }}-${{ steps.setup.outputs.DEP_HASH }}-${{ hashFiles('depends/packages/*') }}
65+
restore-keys: |
66+
${{ runner.os }}-depends-${{ inputs.build-target }}-${{ hashFiles('depends/packages/*') }}
67+
${{ runner.os }}-depends-${{ inputs.build-target }}
68+
69+
- name: Build depends
70+
run: env ${{ steps.setup.outputs.DEP_OPTS }} HOST=${{ steps.setup.outputs.HOST }} make -j$(nproc) -C depends
71+
72+
- name: Save depends cache
73+
uses: actions/cache/save@v4
74+
if: steps.restore.outputs.cache-hit != 'true'
75+
with:
76+
path: |
77+
depends/built
78+
depends/${{ steps.setup.outputs.HOST }}
79+
key: ${{ steps.restore.outputs.cache-primary-key }}

‎.github/workflows/build-src.yml‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
name: Build source
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
build-target:
7+
description: "Target name as defined by inputs.sh"
8+
required: true
9+
type: string
10+
container-path:
11+
description: "Path to built container at registry"
12+
required: true
13+
type: string
14+
depends-key:
15+
description: "Key needed to access cached depends"
16+
required: true
17+
type: string
18+
19+
jobs:
20+
build-src:
21+
name: Build source
22+
runs-on: ubuntu-22.04
23+
container:
24+
image: ${{ inputs.container-path }}
25+
options: --user root
26+
steps:
27+
- name: Checkout code
28+
uses: actions/checkout@v4
29+
with:
30+
ref: ${{ github.event.pull_request.head.sha }}
31+
fetch-depth: 0
32+
33+
- name: Initial setup
34+
id: setup
35+
run: |
36+
git config --global --add advice.detachedHead false
37+
git config --global --add safe.directory "$PWD"
38+
GIT_HEAD="$(git rev-parse HEAD)"
39+
git checkout develop
40+
git checkout "${GIT_HEAD}"
41+
BUILD_TARGET="${{ inputs.build-target }}"
42+
source ./ci/dash/matrix.sh
43+
echo "HOST=${HOST}" >> $GITHUB_OUTPUT
44+
echo "PR_BASE_SHA=${{ github.event.pull_request.base.sha || '' }}" >> $GITHUB_OUTPUT
45+
shell: bash
46+
47+
- name: Restore depends cache
48+
uses: actions/cache/restore@v4
49+
with:
50+
path: |
51+
depends/built
52+
depends/${{ steps.setup.outputs.HOST }}
53+
key: ${{ inputs.depends-key }}
54+
fail-on-cache-miss: true
55+
56+
- name: Manage ccache
57+
uses: actions/cache@v4
58+
with:
59+
path: |
60+
/cache
61+
key: ${{ runner.os }}-${{ inputs.build-target }}-${{ github.sha }}
62+
restore-keys: |
63+
${{ runner.os }}-${{ inputs.build-target }}-${{ github.sha }}
64+
${{ runner.os }}-${{ inputs.build-target }}-${{ steps.setup.outputs.HOST }}
65+
${{ runner.os }}-${{ inputs.build-target }}
66+
67+
- name: Build source and run unit tests
68+
run: |
69+
CCACHE_SIZE="400M"
70+
CACHE_DIR="/cache"
71+
mkdir /output
72+
BASE_OUTDIR="/output"
73+
BUILD_TARGET="${{ inputs.build-target }}"
74+
source ./ci/dash/matrix.sh
75+
./ci/dash/build_src.sh
76+
./ci/dash/test_unittests.sh
77+
shell: bash
78+
79+
- name: Upload build artifacts
80+
uses: actions/upload-artifact@v4
81+
with:
82+
name: build-artifacts-${{ inputs.build-target }}
83+
path: |
84+
/output

0 commit comments

Comments
 (0)