Skip to content

Commit 6c7292d

Browse files
mcgrofmcgrof
committed
bootlinux: add support for A/B kernel testing
Right now we use the same kernel for all target nodes. We want to compare and contrast different kenrels for different features. We add support for A/B testing by leveraging the baseline and dev groups provided to us by KDEVOPS_BASELINE_AND_DEV. This extends the bootlinux playbook by enabling us to allow a different kernel tree / ref to be used for the dev group. This just becomes a configuration thing. The targets are intuitive: make linux # Handles A/B compilation transparently make linux-baseline # Build and install baseline kernel only make linux-dev # Build and install development kernel only We also add a simple check to verify all different build types end up respecting different kernels if we so choose: make check-linux-ab This does not launch targets it just verifies we don't regress in the future with the different ref tags. Generated-by: Claude AI Signed-off-by: Luis Chamberlain <[email protected]>
1 parent 7d2ce95 commit 6c7292d

File tree

18 files changed

+1291
-11
lines changed

18 files changed

+1291
-11
lines changed

‎.github/workflows/linux-ab-testing.yml‎

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
name: Linux A/B Testing Verification
2+
3+
on:
4+
push:
5+
branches:
6+
- '**'
7+
pull_request:
8+
branches:
9+
- '**'
10+
workflow_dispatch: # Allow manual triggering
11+
12+
jobs:
13+
linux-ab-testing-verification:
14+
name: Verify Linux A/B Testing Variables
15+
runs-on: ubuntu-latest
16+
strategy:
17+
matrix:
18+
distro_container:
19+
- debian:testing
20+
- fedora:latest
21+
build_method:
22+
- target
23+
- 9p
24+
- builder
25+
26+
container: ${{ matrix.distro_container }}
27+
steps:
28+
- name: Document test environment
29+
run: |
30+
echo "Running Linux A/B testing verification on ${{ matrix.distro_container }}"
31+
echo "Build method: ${{ matrix.build_method }}"
32+
uname -a
33+
34+
- name: Install dependencies
35+
run: |
36+
if [ "${{ matrix.distro_container }}" = "debian:testing" ]; then
37+
echo "Installing packages for Debian"
38+
apt-get update
39+
apt-get install -y ansible-core make gcc ncurses-dev bison flex git python3
40+
elif [ "${{ matrix.distro_container }}" = "fedora:latest" ]; then
41+
echo "Installing packages for Fedora"
42+
dnf install -y ansible make gcc ncurses-devel bison flex git python3
43+
else
44+
echo "Unknown distribution: ${{ matrix.distro_container }}"
45+
exit 1
46+
fi
47+
48+
- name: Checkout repository
49+
uses: actions/checkout@v4
50+
51+
- name: Configure git for kdevops
52+
run: |
53+
git config --global --add safe.directory '*'
54+
git config --global user.name "kdevops-ci"
55+
git config --global user.email "[email protected]"
56+
57+
- name: Apply A/B testing defconfig
58+
run: |
59+
echo "Applying linux-ab-testing-${{ matrix.build_method }} defconfig"
60+
make defconfig-linux-ab-testing-${{ matrix.build_method }}
61+
62+
# Verify configuration was applied correctly
63+
echo "=== Verifying A/B testing configuration ==="
64+
grep -E "CONFIG_KDEVOPS_BASELINE_AND_DEV=y" .config || exit 1
65+
grep -E "CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y" .config || exit 1
66+
67+
# Check build method specific configs
68+
case "${{ matrix.build_method }}" in
69+
target)
70+
grep -E "CONFIG_BOOTLINUX_TARGET=y" .config || exit 1
71+
;;
72+
9p)
73+
grep -E "CONFIG_BOOTLINUX_9P=y" .config || exit 1
74+
;;
75+
builder)
76+
grep -E "CONFIG_BOOTLINUX_BUILDER=y" .config || exit 1
77+
;;
78+
esac
79+
80+
- name: Run make to generate extra_vars.yaml
81+
run: |
82+
make
83+
84+
- name: Extract and verify kernel references
85+
run: |
86+
echo "=== Extracting kernel references from configuration ==="
87+
88+
# Get the baseline ref (should be master or main)
89+
BASELINE_REF=$(grep "^bootlinux_tree_ref:" extra_vars.yaml | awk '{print $2}')
90+
echo "Baseline ref: $BASELINE_REF"
91+
92+
# Get the dev ref using the inference script
93+
DEV_REF=$(grep "^bootlinux_dev_tree_ref:" extra_vars.yaml | awk '{print $2}')
94+
echo "Dev ref from config: $DEV_REF"
95+
96+
# Since we're in a container without /mirror/linux.git, the script will fallback
97+
# For CI, we'll simulate getting the latest stable tag
98+
if [ -f scripts/infer_last_stable_kernel.sh ]; then
99+
INFERRED_STABLE=$(./scripts/infer_last_stable_kernel.sh 2>/dev/null || echo "v6.12")
100+
echo "Inferred stable version: $INFERRED_STABLE"
101+
fi
102+
103+
# Verify refs are different
104+
if [ "$BASELINE_REF" = "$DEV_REF" ]; then
105+
echo "ERROR: Baseline and dev refs should be different for A/B testing"
106+
exit 1
107+
fi
108+
109+
# Store refs for later verification
110+
echo "BASELINE_REF=$BASELINE_REF" >> $GITHUB_ENV
111+
echo "DEV_REF=$DEV_REF" >> $GITHUB_ENV
112+
113+
- name: Test debug functionality with Ansible
114+
run: |
115+
echo "=== Testing debug output with DEBUG_REF=1 ==="
116+
117+
# Create a minimal hosts file for container testing
118+
cat > hosts << EOF
119+
[all]
120+
localhost ansible_connection=local
121+
122+
[baseline]
123+
localhost ansible_connection=local
124+
125+
[dev]
126+
EOF
127+
128+
# Run the bootlinux playbook with debug enabled and capture output
129+
export DEBUG_REF=1
130+
ansible-playbook -i hosts playbooks/bootlinux.yml --tags vars,debug -v > debug_output.txt 2>&1 || true
131+
132+
# Verify debug output based on build method
133+
case "${{ matrix.build_method }}" in
134+
9p)
135+
echo "=== Verifying 9P debug output (localhost context) ==="
136+
if grep -q "active_linux_ref" debug_output.txt; then
137+
echo "✓ Found active_linux_ref in 9P debug output"
138+
else
139+
echo "✗ Missing active_linux_ref in 9P debug output"
140+
cat debug_output.txt
141+
exit 1
142+
fi
143+
;;
144+
target|builder)
145+
echo "=== Verifying non-9P debug output (per-node context) ==="
146+
if grep -q "target_linux_ref" debug_output.txt; then
147+
echo "✓ Found target_linux_ref in non-9P debug output"
148+
else
149+
echo "✗ Missing target_linux_ref in non-9P debug output"
150+
cat debug_output.txt
151+
exit 1
152+
fi
153+
;;
154+
esac
155+
156+
- name: Verify A/B testing Makefile rules
157+
run: |
158+
echo "=== Verifying A/B testing Makefile structure ==="
159+
160+
# Check that linux target depends on linux-baseline and linux-dev
161+
if grep -A5 "^linux:" workflows/linux/Makefile | grep -q "linux-baseline linux-dev"; then
162+
echo "✓ Makefile has correct A/B testing dependencies"
163+
else
164+
echo "✗ Makefile missing A/B testing dependencies"
165+
exit 1
166+
fi
167+
168+
# Verify linux-baseline and linux-dev targets exist
169+
if grep -q "^linux-baseline:" workflows/linux/Makefile && \
170+
grep -q "^linux-dev:" workflows/linux/Makefile; then
171+
echo "✓ Both linux-baseline and linux-dev targets exist"
172+
else
173+
echo "✗ Missing linux-baseline or linux-dev targets"
174+
exit 1
175+
fi
176+
177+
- name: Test variable resolution patterns
178+
run: |
179+
echo "=== Testing variable resolution patterns ==="
180+
181+
# Create test playbook to verify variable resolution
182+
cat > test_vars.yml << 'EOF'
183+
---
184+
- hosts: localhost
185+
connection: local
186+
tasks:
187+
- name: Load extra vars
188+
include_vars: extra_vars.yaml
189+
190+
- name: Display loaded variables
191+
debug:
192+
msg: |
193+
bootlinux_tree_ref: {{ bootlinux_tree_ref | default('undefined') }}
194+
bootlinux_dev_tree_ref: {{ bootlinux_dev_tree_ref | default('undefined') }}
195+
kdevops_baseline_and_dev: {{ kdevops_baseline_and_dev | default(false) }}
196+
bootlinux_ab_different_ref: {{ bootlinux_ab_different_ref | default(false) }}
197+
EOF
198+
199+
ansible-playbook test_vars.yml -v
200+
201+
- name: Summary report
202+
if: always()
203+
run: |
204+
echo "=== A/B Testing Verification Summary ==="
205+
echo "Distribution: ${{ matrix.distro_container }}"
206+
echo "Build Method: ${{ matrix.build_method }}"
207+
echo "Baseline Ref: ${BASELINE_REF:-not set}"
208+
echo "Dev Ref: ${DEV_REF:-not set}"
209+
echo ""
210+
211+
if [ -f .config ]; then
212+
echo "Key configurations:"
213+
grep -E "(BASELINE_AND_DEV|AB_DIFFERENT_REF|BOOTLINUX_TARGET|BOOTLINUX_9P|BOOTLINUX_BUILDER)" .config | head -10
214+
fi
215+
216+
echo ""
217+
echo "Test completed successfully ✓"

‎.github/workflows/linux-ab.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
name: Run kdevops linux-ab tests on self-hosted runner
2+
3+
on:
4+
push:
5+
branches:
6+
- '**'
7+
pull_request:
8+
branches:
9+
- '**'
10+
workflow_dispatch: # Add this for manual triggering of the workflow
11+
12+
jobs:
13+
run-kdevops:
14+
name: Run kdevops CI
15+
runs-on: [self-hosted, Linux, X64]
16+
steps:
17+
- name: Checkout repository
18+
uses: actions/checkout@v4
19+
20+
- name: Set CI metadata for kdevops-results-archive
21+
run: |
22+
echo "$(basename ${{ github.repository }})" > ci.trigger
23+
git log -1 --pretty=format:"%s" > ci.subject
24+
# Start out pessimistic
25+
echo "not ok" > ci.result
26+
echo "Nothing to write home about." > ci.commit_extra
27+
28+
- name: Set kdevops path
29+
run: echo "KDEVOPS_PATH=$GITHUB_WORKSPACE" >> $GITHUB_ENV
30+
31+
- name: Configure git
32+
run: |
33+
git config --global --add safe.directory '*'
34+
git config --global user.name "kdevops"
35+
git config --global user.email "[email protected]"
36+
37+
- name: Run kdevops check-linux-ab
38+
run: |
39+
make check-linux-ab
40+
echo "ok" > ci.result
41+
42+
# Ensure make destroy always runs, even on failure
43+
- name: Run kdevops make destroy
44+
if: always() # This ensures the step runs even if previous steps failed
45+
run: |
46+
make destroy
47+
make mrproper

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -243,6 +243,7 @@ $(KDEVOPS_NODES): .config $(ANSIBLE_CFG_FILE) $(KDEVOPS_NODES_TEMPLATE)
243243
DEFAULT_DEPS += $(LOCALHOST_SETUP_WORK)
244244

245245
include scripts/tests.Makefile
246+
include scripts/linux-ab-testing.Makefile
246247
include scripts/ci.Makefile
247248
include scripts/archive.Makefile
248249
include scripts/defconfig.Makefile

‎PROMPTS.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,3 +123,55 @@ source "workflows/mmtests/Kconfig.thpchallenge"
123123
source "workflows/mmtests/Kconfig.fs"
124124

125125
This separation is preferred as it helps us scale.
126+
127+
## Kernel development and A/B testing support
128+
129+
### Adding A/B kernel testing support for different kernel versions
130+
131+
**Prompt:**
132+
We want to add support for when users enable KDEVOPS_BASELINE_AND_DEV we want
133+
to extend workflows/linux/Kconfig with the a choise set of options to either a)
134+
use the same kernel ref or b) allow the user to specify a different ref tag.
135+
This will enable A/B testing with different kernel versions. When a different
136+
kernel refs are desirable we will want to extend the compilation step and
137+
installation of the Linux kernel in two steps. The first will be for the ref
138+
and target of A (baseline tag) and the second will be for the target ref of B
139+
(dev tag). However we want to fold these two steps in one for when
140+
KDEVOPS_BASELINE_AND_DEV is used and make install is used, it would happen
141+
transparently for us. The resulting linux kernel directory would end up with
142+
the "dev" ref at the end. In case a user wants to re-compile a target ref for
143+
baseline or dev we want to add (if we don't have already) a make linux-baseline
144+
and make linux-dev so that we can build and install the target ref tag on the
145+
baseline (A) or dev (B). The make linux target then would serially do make
146+
linux-baseline and make linux-dev. Extend documentation for all this and also
147+
add the respective prompt to PROMPTS.md once done. Avoid adding extra spaces to
148+
code or documentation at the end of each line. These end up in red color on
149+
diffs and hurt my eyes. Extend CLAUDE.md to understand styling for these rules
150+
about not wanting lines ending in white space for styling.
151+
152+
**AI:** Claude Code
153+
**Commit:** [To be determined]
154+
**Result:** Complete A/B kernel testing implementation with comprehensive configuration options.
155+
**Grading:** 70%
156+
157+
**Notes:**
158+
159+
The implementation successfully added:
160+
161+
1. **Makefile Implementation**: the AI failed to grasp the value of
162+
output yaml, and made ugly Makefile changes to extract variables.
163+
164+
2. **Ansible Integration**: The AI failed to write the required changes on
165+
the ansible playbook at first. A secondary prompt made it just move the
166+
definitions to the ansible playbook but failed to address serially compiling
167+
linux for the baseline group first followed by the dev group after.
168+
169+
3. **Documentation**: The AI is not grasping the preference to respect 80
170+
character lengths.
171+
172+
4. **Issues**: The AI failed to understand a really obscure lesson which even
173+
humans have issues in understanding in ansible, you can't override a fact
174+
and later use it specially if being used on multple hosts. The best thing
175+
to do is to use a separate fact if you want a true dynamic variable. This
176+
is why we switched to an active ref prefix for the baseline and dev group
177+
ref tags.

‎defconfigs/linux-ab-testing‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
CONFIG_GUESTFS=y
2+
CONFIG_LIBVIRT=y
3+
4+
CONFIG_WORKFLOWS=y
5+
CONFIG_WORKFLOW_LINUX_CUSTOM=y
6+
7+
CONFIG_BOOTLINUX=y
8+
CONFIG_BOOTLINUX_9P=y
9+
10+
# Enable baseline and dev testing
11+
CONFIG_KDEVOPS_BASELINE_AND_DEV=y
12+
13+
# Enable A/B testing with different kernel references
14+
CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y

‎defconfigs/linux-ab-testing-9p‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
CONFIG_GUESTFS=y
2+
CONFIG_LIBVIRT=y
3+
4+
CONFIG_WORKFLOWS=y
5+
CONFIG_WORKFLOW_LINUX_CUSTOM=y
6+
7+
CONFIG_BOOTLINUX=y
8+
CONFIG_BOOTLINUX_9P=y
9+
# CONFIG_BOOTLINUX_BUILDER is not set
10+
11+
# Enable baseline and dev testing
12+
CONFIG_KDEVOPS_BASELINE_AND_DEV=y
13+
14+
# Enable A/B testing with different kernel references
15+
CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y

‎defconfigs/linux-ab-testing-builder‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
CONFIG_GUESTFS=y
2+
CONFIG_LIBVIRT=y
3+
4+
CONFIG_WORKFLOWS=y
5+
CONFIG_WORKFLOW_LINUX_CUSTOM=y
6+
7+
CONFIG_BOOTLINUX=y
8+
# CONFIG_BOOTLINUX_9P is not set
9+
CONFIG_BOOTLINUX_BUILDER=y
10+
11+
# Enable baseline and dev testing
12+
CONFIG_KDEVOPS_BASELINE_AND_DEV=y
13+
14+
# Enable A/B testing with different kernel references
15+
CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y

‎defconfigs/linux-ab-testing-target‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
CONFIG_GUESTFS=y
2+
CONFIG_LIBVIRT=y
3+
4+
CONFIG_WORKFLOWS=y
5+
CONFIG_WORKFLOW_LINUX_CUSTOM=y
6+
7+
CONFIG_BOOTLINUX=y
8+
# CONFIG_BOOTLINUX_9P is not set
9+
# CONFIG_BOOTLINUX_BUILDER is not set
10+
11+
# Enable baseline and dev testing
12+
CONFIG_KDEVOPS_BASELINE_AND_DEV=y
13+
14+
# Enable A/B testing with different kernel references
15+
CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y

0 commit comments

Comments
 (0)