Skip to content

Commit 4dd7684

Browse files
dkrucesmcgrof
dkruces
authored and
mcgrof
committed
github: refactor into reusable actions and workflows
Refactor the .github/ CI support into reusable actions and workflows. This allows to use the new actions not only internally in kdevops but also in other projects such linux-modules-kpd and similar. Keeping the ci logic in one place and allowing other projects to get the latest changes. CI supports the following: - Split workflow in setup, test, archive and cleanup actions - Fixed kernel: v6.15 on linux tree. This is to speed up builds relying on the same kernel. We can update when needed or timely basis - Fixed workflows: blktests_nvme and xfs_reflink_4k. We want to validate kdevops workflows and blktests with nvme section/profile has been added on top of the current fstests/xfs reflink 4k - 2 CI modes: A. kdevops selftest/ci validation. A single test execution for the given workflow (e.g. generic/003 for xfs). This has it's own commit format. B. kdevops full workflow. This is to validate a specific kernel on a given test suite. It's supported on manual trigger as well as on schedule. Add automatic runner assignment based on ci_workflow names to enable parallel execution on dedicated runners: - Parallelize GitHub workflow execution on dedicated runners based on CI_WORKFLOW names (using tags). E.g.: * blktests workflows → [self-hosted, Linux, X64, blktests] * xfs workflows → [self-hosted, Linux, X64, xfs] - Set timeout 120 minutes for CI B mode. - Disable fail-fast to matrix strategy in both push.yml and schedule.yml workflows to prevent cancellation of successful jobs when other matrix jobs fail. Example: this allows blktests_nvme to continue running even if xfs_reflink_4k fails Signed-off-by: Daniel Gomez <[email protected]> Signed-off-by: Luis Chamberlain <[email protected]>
1 parent 19b93c8 commit 4dd7684

File tree

14 files changed

+1214
-99
lines changed

14 files changed

+1214
-99
lines changed

‎.github/actions/archive/action.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# SPDX-License-Identifier: GPL-2.0
2+
---
3+
name: Archive results
4+
description: Archive kdevops results in https://github.com/linux-kdevops/kdevops-results-archive.git
5+
inputs:
6+
ci_workflow:
7+
required: false
8+
type: string
9+
default: 'demo'
10+
dir:
11+
description: 'Directory'
12+
required: true
13+
default: 'workdir'
14+
15+
runs:
16+
using: "composite"
17+
steps:
18+
- name: Get systemd journal files
19+
working-directory: ${{ inputs.dir }}/kdevops
20+
shell: bash
21+
run: |
22+
make journal-dump
23+
24+
- name: Build our kdevops archive results
25+
working-directory: ${{ inputs.dir }}/kdevops
26+
shell: bash
27+
run: |
28+
make ci-archive CI_WORKFLOW="${{ inputs.ci_workflow }}"

‎.github/actions/cleanup/action.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# SPDX-License-Identifier: GPL-2.0
2+
---
3+
name: Cleanup kdevops VMs
4+
description: Destroy VMs and cleanup workspace
5+
6+
inputs:
7+
dir:
8+
description: 'Directory'
9+
required: true
10+
default: 'workdir'
11+
12+
runs:
13+
using: "composite"
14+
steps:
15+
- name: Run kdevops make destroy
16+
working-directory: ${{ inputs.dir }}/kdevops
17+
shell: bash
18+
run: |
19+
make destroy
20+
21+
- name: Remove working-directory
22+
shell: bash
23+
run: |
24+
rm --recursive --force --verbose ${{ inputs.dir }}

‎.github/actions/setup/action.yml‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
# SPDX-License-Identifier: GPL-2.0
2+
---
3+
name: Setup kdevops
4+
description: Setup kdevops workspace
5+
6+
inputs:
7+
ci_workflow:
8+
required: false
9+
type: string
10+
default: 'demo'
11+
dir:
12+
description: 'Directory'
13+
required: true
14+
default: 'workdir'
15+
kernel_tree:
16+
required: false
17+
type: string
18+
default: 'linux'
19+
kernel_ref:
20+
required: false
21+
type: string
22+
default: 'master'
23+
24+
runs:
25+
using: "composite"
26+
steps:
27+
- name: Create workspace directory
28+
shell: bash
29+
run: |
30+
set -euxo pipefail
31+
rm --recursive --force --verbose ${{ inputs.dir }}
32+
mkdir --parent --verbose ${{ inputs.dir }}
33+
34+
- name: Configure git
35+
shell: bash
36+
run: |
37+
set -euxo pipefail
38+
git config --global --add safe.directory '*'
39+
git config --global user.name "kdevops"
40+
git config --global user.email "[email protected]"
41+
42+
- name: Checkout kdevops
43+
working-directory: ${{ inputs.dir }}
44+
shell: bash
45+
run: |
46+
set -euxo pipefail
47+
git clone https://github.com/dkruces/kdevops.git --branch main kdevops
48+
49+
- name: Checkout custom branch with delta on kdevops/linux
50+
working-directory: ${{ inputs.dir }}/kdevops
51+
shell: bash
52+
run: |
53+
set -euxo pipefail
54+
LINUX_TREE="/mirror/${{ inputs.kernel_tree }}.git"
55+
LINUX_TREE_REF="${{ inputs.kernel_ref }}"
56+
git clone $LINUX_TREE linux
57+
cd linux
58+
git checkout $LINUX_TREE_REF
59+
git log -1
60+
61+
- name: Make sure our repo kdevops defconfig exists
62+
id: defconfig
63+
working-directory: ${{ inputs.dir }}/kdevops
64+
shell: bash
65+
run: |
66+
set -euxo pipefail
67+
KDEVOPS_DEFCONFIG=${{ inputs.ci_workflow }}
68+
69+
if [[ ! -f defconfigs/$KDEVOPS_DEFCONFIG ]]; then
70+
echo "Missing defconfig: defconfigs/$KDEVOPS_DEFCONFIG"
71+
exit 1
72+
fi
73+
74+
"${{ github.workspace }}/scripts/github_output.sh" \
75+
KDEVOPS_DEFCONFIG "$KDEVOPS_DEFCONFIG"
76+
77+
- name: Initialize CI metadata for kdevops-results-archive for linux
78+
id: metadata
79+
working-directory: ${{ inputs.dir }}/kdevops/linux
80+
shell: bash
81+
run: |
82+
set -euxo pipefail
83+
echo "${{ inputs.kernel_tree }}" > ../ci.trigger
84+
# Get the kdevops commit subject, not kernel commit
85+
cd "${{ github.workspace }}" && git log -1 --pretty=format:"%s" > "${{ inputs.dir }}/kdevops/ci.subject"
86+
cd - > /dev/null
87+
echo "${{ inputs.kernel_ref }}" > ../ci.ref
88+
89+
RELEVANT_GIT_TAG=$(cat ../ci.ref)
90+
RELEVANT_GIT_REF=$(git rev-parse --short=12 $RELEVANT_GIT_TAG)
91+
92+
"${{ github.workspace }}/scripts/github_output.sh" \
93+
LINUX_GIT_REF "$RELEVANT_GIT_REF"
94+
"${{ github.workspace }}/scripts/github_output.sh" \
95+
LINUX_GIT_TAG "$RELEVANT_GIT_TAG"
96+
97+
# Start out pessimistic
98+
echo "unknown" > ../ci.result
99+
echo "Nothing to write home about." > ../ci.commit_extra
100+
101+
- name: Run kdevops make defconfig-repo
102+
working-directory: ${{ inputs.dir }}/kdevops
103+
env:
104+
LINUX_GIT_TAG: ${{ steps.metadata.outputs.LINUX_GIT_TAG }}
105+
LINUX_GIT_REF: ${{ steps.metadata.outputs.LINUX_GIT_REF }}
106+
KDEVOPS_DEFCONFIG: ${{ steps.defconfig.outputs.KDEVOPS_DEFCONFIG }}
107+
shell: bash
108+
run: |
109+
set -euxo pipefail
110+
LINUX_TREE="/mirror/${{ inputs.kernel_tree }}.git"
111+
LINUX_TREE_REF="$LINUX_GIT_TAG"
112+
113+
# We make the compromise here to use a relevant git tag for the
114+
# host prefix so that folks can easily tell what exact kernel tree
115+
# is being tested by using the relevant git ref. That is, if you
116+
# pushed a tree with the .github/ directory as the top of the tree,
117+
# that commit will not be used, we'll use the last one as that is
118+
# the relevant git ref we want to annotate a test for.
119+
#
120+
# The compromise here is that we expect no two same identical tests
121+
# on the same self-hosted server. We could extend this with something
122+
# like github.run_id but its not yet clear if we can have kdevops
123+
# hosts with a bundled prefix ID like that ref-runid-testname. Tests
124+
# have been done with the full lenght sha1sum though and we know that
125+
# does work.
126+
KDEVOPS_HOSTS_PREFIX="kci-$LINUX_GIT_REF"
127+
128+
echo "Going to use defconfig-$KDEVOPS_DEFCONFIG"
129+
130+
echo "Linux tree: $LINUX_TREE"
131+
echo "Linux trigger ref: $LINUX_TREE_REF"
132+
echo "Linux tag: $LINUX_GIT_TAG"
133+
echo "Runner ID: ${{ github.run_id }}"
134+
echo "kdevops host prefix: $KDEVOPS_HOSTS_PREFIX"
135+
echo "kdevops defconfig: defconfig-$KDEVOPS_DEFCONFIG"
136+
137+
# Customize KMOD_TIMEOUT when required
138+
KMOD_TIMEOUT_ARG=
139+
if [[ "$(hostname)" == *smc111* && \
140+
"$KDEVOPS_DEFCONFIG" == "linux-modules-kpd" ]]; then
141+
KMOD_TIMEOUT_ARG="KMOD_TIMEOUT=222"
142+
fi
143+
144+
KDEVOPS_ARGS="\
145+
KDEVOPS_HOSTS_PREFIX=$KDEVOPS_HOSTS_PREFIX \
146+
LINUX_TREE=$LINUX_TREE \
147+
LINUX_TREE_REF=$LINUX_TREE_REF \
148+
${KMOD_TIMEOUT_ARG} \
149+
defconfig-$KDEVOPS_DEFCONFIG"
150+
echo "Going to run:"
151+
echo "make $KDEVOPS_ARGS"
152+
153+
make $KDEVOPS_ARGS
154+
./scripts/kconfig/merge_config.sh \
155+
-n .config \
156+
defconfigs/configs/diy.config \
157+
defconfigs/configs/ci.config
158+
159+
- name: Run kdevops make
160+
working-directory: ${{ inputs.dir }}/kdevops
161+
shell: bash
162+
run: |
163+
set -euxo pipefail
164+
make -j$(nproc)
165+
166+
- name: Run kdevops make bringup
167+
working-directory: ${{ inputs.dir }}/kdevops
168+
shell: bash
169+
run: |
170+
set -euxo pipefail
171+
ls -ld linux
172+
make destroy
173+
make bringup
174+
175+
- name: Build linux and boot test nodes on test kernel
176+
working-directory: ${{ inputs.dir }}/kdevops
177+
shell: bash
178+
run: |
179+
set -euxo pipefail
180+
make linux
181+
182+
- name: Build required ci tests
183+
working-directory: ${{ inputs.dir }}/kdevops
184+
shell: bash
185+
run: |
186+
set -euxo pipefail
187+
make ci-build-test CI_WORKFLOW=${{ inputs.ci_workflow }}

0 commit comments

Comments
 (0)