Skip to content

Commit 7f25db7

Browse files
mcgrofmcgrof
committed
bootlinux: add support for A/B kernel testing
Right now we use the same kernel for all target nodes. We want to compare and contrast different kenrels for different features. We add support for A/B testing by leveraging the baseline and dev groups provided to us by KDEVOPS_BASELINE_AND_DEV. This extends the bootlinux playbook by enabling us to allow a different kernel tree / ref to be used for the dev group. This just becomes a configuration thing. The targets are intuitive: make linux # Handles A/B compilation transparently make linux-baseline # Build and install baseline kernel only make linux-dev # Build and install development kernel only We also add a simple check to verify all different build types end up respecting different kernels if we so choose: make check-linux-ab This does not launch targets it just verifies we don't regress in the future with the different ref tags. Generated-by: Claude AI Signed-off-by: Luis Chamberlain <[email protected]>
1 parent 7d2ce95 commit 7f25db7

File tree

17 files changed

+1074
-11
lines changed

17 files changed

+1074
-11
lines changed

‎.github/workflows/linux-ab.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
name: Run kdevops linux-ab tests on self-hosted runner
2+
3+
on:
4+
push:
5+
branches:
6+
- '**'
7+
pull_request:
8+
branches:
9+
- '**'
10+
workflow_dispatch: # Add this for manual triggering of the workflow
11+
12+
jobs:
13+
run-kdevops:
14+
name: Run kdevops CI
15+
runs-on: [self-hosted, Linux, X64]
16+
steps:
17+
- name: Checkout repository
18+
uses: actions/checkout@v4
19+
20+
- name: Set CI metadata for kdevops-results-archive
21+
run: |
22+
echo "$(basename ${{ github.repository }})" > ci.trigger
23+
git log -1 --pretty=format:"%s" > ci.subject
24+
# Start out pessimistic
25+
echo "not ok" > ci.result
26+
echo "Nothing to write home about." > ci.commit_extra
27+
28+
- name: Set kdevops path
29+
run: echo "KDEVOPS_PATH=$GITHUB_WORKSPACE" >> $GITHUB_ENV
30+
31+
- name: Configure git
32+
run: |
33+
git config --global --add safe.directory '*'
34+
git config --global user.name "kdevops"
35+
git config --global user.email "[email protected]"
36+
37+
- name: Run kdevops check-linux-ab
38+
run: |
39+
make check-linux-ab
40+
echo "ok" > ci.result
41+
42+
# Ensure make destroy always runs, even on failure
43+
- name: Run kdevops make destroy
44+
if: always() # This ensures the step runs even if previous steps failed
45+
run: |
46+
make destroy
47+
make mrproper

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -243,6 +243,7 @@ $(KDEVOPS_NODES): .config $(ANSIBLE_CFG_FILE) $(KDEVOPS_NODES_TEMPLATE)
243243
DEFAULT_DEPS += $(LOCALHOST_SETUP_WORK)
244244

245245
include scripts/tests.Makefile
246+
include scripts/linux-ab-testing.Makefile
246247
include scripts/ci.Makefile
247248
include scripts/archive.Makefile
248249
include scripts/defconfig.Makefile

‎PROMPTS.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,3 +123,55 @@ source "workflows/mmtests/Kconfig.thpchallenge"
123123
source "workflows/mmtests/Kconfig.fs"
124124

125125
This separation is preferred as it helps us scale.
126+
127+
## Kernel development and A/B testing support
128+
129+
### Adding A/B kernel testing support for different kernel versions
130+
131+
**Prompt:**
132+
We want to add support for when users enable KDEVOPS_BASELINE_AND_DEV we want
133+
to extend workflows/linux/Kconfig with the a choise set of options to either a)
134+
use the same kernel ref or b) allow the user to specify a different ref tag.
135+
This will enable A/B testing with different kernel versions. When a different
136+
kernel refs are desirable we will want to extend the compilation step and
137+
installation of the Linux kernel in two steps. The first will be for the ref
138+
and target of A (baseline tag) and the second will be for the target ref of B
139+
(dev tag). However we want to fold these two steps in one for when
140+
KDEVOPS_BASELINE_AND_DEV is used and make install is used, it would happen
141+
transparently for us. The resulting linux kernel directory would end up with
142+
the "dev" ref at the end. In case a user wants to re-compile a target ref for
143+
baseline or dev we want to add (if we don't have already) a make linux-baseline
144+
and make linux-dev so that we can build and install the target ref tag on the
145+
baseline (A) or dev (B). The make linux target then would serially do make
146+
linux-baseline and make linux-dev. Extend documentation for all this and also
147+
add the respective prompt to PROMPTS.md once done. Avoid adding extra spaces to
148+
code or documentation at the end of each line. These end up in red color on
149+
diffs and hurt my eyes. Extend CLAUDE.md to understand styling for these rules
150+
about not wanting lines ending in white space for styling.
151+
152+
**AI:** Claude Code
153+
**Commit:** [To be determined]
154+
**Result:** Complete A/B kernel testing implementation with comprehensive configuration options.
155+
**Grading:** 70%
156+
157+
**Notes:**
158+
159+
The implementation successfully added:
160+
161+
1. **Makefile Implementation**: the AI failed to grasp the value of
162+
output yaml, and made ugly Makefile changes to extract variables.
163+
164+
2. **Ansible Integration**: The AI failed to write the required changes on
165+
the ansible playbook at first. A secondary prompt made it just move the
166+
definitions to the ansible playbook but failed to address serially compiling
167+
linux for the baseline group first followed by the dev group after.
168+
169+
3. **Documentation**: The AI is not grasping the preference to respect 80
170+
character lengths.
171+
172+
4. **Issues**: The AI failed to understand a really obscure lesson which even
173+
humans have issues in understanding in ansible, you can't override a fact
174+
and later use it specially if being used on multple hosts. The best thing
175+
to do is to use a separate fact if you want a true dynamic variable. This
176+
is why we switched to an active ref prefix for the baseline and dev group
177+
ref tags.

‎defconfigs/linux-ab-testing‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
CONFIG_GUESTFS=y
2+
CONFIG_LIBVIRT=y
3+
4+
CONFIG_WORKFLOWS=y
5+
CONFIG_WORKFLOW_LINUX_CUSTOM=y
6+
7+
CONFIG_BOOTLINUX=y
8+
CONFIG_BOOTLINUX_9P=y
9+
10+
# Enable baseline and dev testing
11+
CONFIG_KDEVOPS_BASELINE_AND_DEV=y
12+
13+
# Enable A/B testing with different kernel references
14+
CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y

‎defconfigs/linux-ab-testing-9p‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
CONFIG_GUESTFS=y
2+
CONFIG_LIBVIRT=y
3+
4+
CONFIG_WORKFLOWS=y
5+
CONFIG_WORKFLOW_LINUX_CUSTOM=y
6+
7+
CONFIG_BOOTLINUX=y
8+
CONFIG_BOOTLINUX_9P=y
9+
# CONFIG_BOOTLINUX_BUILDER is not set
10+
11+
# Enable baseline and dev testing
12+
CONFIG_KDEVOPS_BASELINE_AND_DEV=y
13+
14+
# Enable A/B testing with different kernel references
15+
CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y

‎defconfigs/linux-ab-testing-builder‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
CONFIG_GUESTFS=y
2+
CONFIG_LIBVIRT=y
3+
4+
CONFIG_WORKFLOWS=y
5+
CONFIG_WORKFLOW_LINUX_CUSTOM=y
6+
7+
CONFIG_BOOTLINUX=y
8+
# CONFIG_BOOTLINUX_9P is not set
9+
CONFIG_BOOTLINUX_BUILDER=y
10+
11+
# Enable baseline and dev testing
12+
CONFIG_KDEVOPS_BASELINE_AND_DEV=y
13+
14+
# Enable A/B testing with different kernel references
15+
CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y

‎defconfigs/linux-ab-testing-target‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
CONFIG_GUESTFS=y
2+
CONFIG_LIBVIRT=y
3+
4+
CONFIG_WORKFLOWS=y
5+
CONFIG_WORKFLOW_LINUX_CUSTOM=y
6+
7+
CONFIG_BOOTLINUX=y
8+
# CONFIG_BOOTLINUX_9P is not set
9+
# CONFIG_BOOTLINUX_BUILDER is not set
10+
11+
# Enable baseline and dev testing
12+
CONFIG_KDEVOPS_BASELINE_AND_DEV=y
13+
14+
# Enable A/B testing with different kernel references
15+
CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y

‎docs/kdevops-make-linux.md‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,161 @@ To verify the kernel on it:
1313
```bash
1414
make uname
1515
```
16+
17+
## A/B Kernel Testing
18+
19+
kdevops supports A/B testing with different kernel versions when
20+
`KDEVOPS_BASELINE_AND_DEV` is enabled. This allows you to compare performance
21+
or behavior between different kernel versions across baseline and development nodes.
22+
23+
### Configuration Options
24+
25+
When A/B testing is enabled, you can choose between two approaches:
26+
27+
#### Same Kernel Reference (Default)
28+
Use the same kernel tree and reference for both baseline and dev nodes:
29+
```
30+
A/B kernel testing configuration (BOOTLINUX_AB_SAME_REF) [Y/n/?]
31+
```
32+
33+
This is useful for testing configuration changes or different test parameters
34+
with identical kernels.
35+
36+
#### Different Kernel References
37+
Use different kernel references for baseline and dev nodes:
38+
```
39+
A/B kernel testing configuration
40+
1. Use same kernel reference for baseline and dev (BOOTLINUX_AB_SAME_REF)
41+
> 2. Use different kernel references for baseline and dev (BOOTLINUX_AB_DIFFERENT_REF)
42+
```
43+
44+
This enables testing between different kernel versions, commits, or branches.
45+
46+
When using different references, configure:
47+
- **Development kernel tree URL**: Git repository (defaults to baseline tree)
48+
- **Development kernel reference**: Branch, tag, or commit (e.g., "v6.8", "linux-next")
49+
- **Development kernel release/local version**: Custom version strings for identification
50+
51+
### Make Targets
52+
53+
#### Standard Linux Building
54+
```bash
55+
make linux # Build and install kernels for all nodes
56+
```
57+
58+
When A/B testing with different references is enabled, this automatically:
59+
1. Builds and installs baseline kernel on baseline nodes
60+
2. Builds and installs development kernel on dev nodes
61+
3. Leaves the working directory with the dev kernel checked out
62+
63+
#### Individual Node Targeting
64+
```bash
65+
make linux-baseline # Build and install kernel for baseline nodes only
66+
make linux-dev # Build and install kernel for dev nodes only
67+
```
68+
69+
These targets are available when `KDEVOPS_BASELINE_AND_DEV=y` and allow
70+
selective building and installation.
71+
72+
### Usage Examples
73+
74+
#### Testing Kernel Versions
75+
Compare v6.7 (baseline) vs v6.8 (development):
76+
77+
```bash
78+
# Configure baseline kernel
79+
menuconfig → Workflows → Linux kernel → Git tree to clone: linus
80+
Reference to use: v6.7
81+
82+
# Configure A/B testing
83+
menuconfig → Workflows → Linux kernel → A/B kernel testing
84+
→ Use different kernel references
85+
→ Development kernel reference: v6.8
86+
87+
make bringup # Provision baseline and dev nodes
88+
make linux # Install v6.7 on baseline, v6.8 on dev
89+
make fstests # Run tests on both kernel versions
90+
make fstests-compare # Compare results between versions
91+
```
92+
93+
#### Testing Development Branches
94+
Compare stable vs linux-next:
95+
96+
```bash
97+
# Baseline: stable kernel
98+
menuconfig → Reference to use: v6.8
99+
100+
# Development: linux-next
101+
menuconfig → A/B kernel testing → Development kernel reference: linux-next
102+
103+
make linux-baseline # Install stable kernel on baseline nodes
104+
make linux-dev # Install linux-next on dev nodes
105+
```
106+
107+
#### Bisection Support
108+
Test specific commits during bisection:
109+
110+
```bash
111+
# Update development reference for bisection
112+
menuconfig → Development kernel reference: abc123def
113+
114+
make linux-dev # Install bisection commit on dev nodes
115+
# Run tests and analyze results
116+
```
117+
118+
### Working Directory State
119+
120+
After running `make linux` with different references:
121+
- The Linux source directory contains the **development kernel** checkout
122+
- Both baseline and dev nodes have their respective kernels installed
123+
- Use `git log --oneline -5` to verify the current checkout
124+
125+
To switch the working directory to baseline:
126+
```bash
127+
git checkout v6.7 # Switch to baseline reference
128+
```
129+
130+
### Integration with Testing Workflows
131+
132+
A/B kernel testing integrates seamlessly with all kdevops testing workflows:
133+
134+
```bash
135+
# Run fstests with kernel comparison
136+
make linux # Install different kernels
137+
make fstests # Test both kernel versions
138+
make fstests-compare # Generate comparison analysis
139+
140+
# Run fio-tests with kernel comparison
141+
make linux # Install different kernels
142+
make fio-tests # Performance test both kernels
143+
make fio-tests-compare # Compare performance metrics
144+
145+
# Run sysbench with kernel comparison
146+
make linux # Install different kernels
147+
make sysbench # Database tests on both kernels
148+
```
149+
150+
### Best Practices
151+
152+
1. **Version Identification**: Use descriptive kernel release versions to distinguish builds
153+
2. **Sequential Testing**: Install kernels before running test workflows
154+
3. **Result Organization**: Use baseline/dev labels in test result analysis
155+
4. **Git Management**: Keep track of which reference is currently checked out
156+
5. **Systematic Comparison**: Use `*-compare` targets for meaningful analysis
157+
158+
### Troubleshooting
159+
160+
#### Build Failures
161+
- Ensure both kernel references are valid and accessible
162+
- Check that build dependencies are installed on all nodes
163+
- Verify git repository permissions and network connectivity
164+
165+
#### Version Conflicts
166+
- Use different `kernelrelease` and `localversion` settings for clear identification
167+
- Check `/boot` directory for kernel installation conflicts
168+
- Verify GRUB configuration after kernel installation
169+
170+
#### Node Targeting Issues
171+
- Confirm `KDEVOPS_BASELINE_AND_DEV=y` is enabled
172+
- Verify baseline and dev node groups exist in inventory
173+
- Check ansible host patterns with `make linux-baseline HOSTS=baseline`

‎playbooks/roles/bootlinux/defaults/main.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,3 +52,17 @@ bootlinux_tree_set_by_cli: False
5252
bootlinux_artifacts_dir: "{{ topdir_path }}/workflows/linux/artifacts"
5353
kernel_packages: []
5454
workflow_linux_packaged: false
55+
56+
# A/B testing defaults
57+
bootlinux_ab_same_ref: True
58+
bootlinux_ab_different_ref: False
59+
60+
# Development kernel settings (used when bootlinux_ab_different_ref is True)
61+
bootlinux_dev_tree: ""
62+
target_linux_dev_ref: "master"
63+
target_linux_dev_kernelrelease: ""
64+
target_linux_dev_localversion: ""
65+
bootlinux_tree_custom_kernelrelease: False
66+
bootlinux_tree_custom_localversion: false
67+
bootlinux_is_dev_node: False
68+
bootlinux_debug_ref: "{{ lookup('env', 'DEBUG_REF') | default(false, true) | bool }}"

0 commit comments

Comments
 (0)