test: Add integration test running on github runner #1496

henrywang · 2025-08-07T15:27:49Z

New integration test workflow:

Run integration test in Github runner or run locally with TEST_OS=fedora-42 make test
build job:

build bootc binary in container and replace the bootc inside the bootc image
bootc install to-disk to generate disk image
archive disk file (skip this step if run locally), one image can be used by the test jobs in parallel

test job:

download disk image from archive
boot disk image with qemu-system
run tmt test against this VM

gemini-code-assist

Code Review

This pull request introduces a new integration test running on GitHub runners, which is a great addition. The implementation involves new scripts for building a test image and running it in QEMU. My review focuses on improving the robustness and security of these new test scripts. I've identified a few critical issues that would prevent the tests from running as intended, such as an undefined command and incorrect error handling in the presence of set -e. I've also pointed out security concerns regarding disabled TLS verification and opportunities to make the scripts more robust and maintainable. Please see the detailed comments for suggestions.

tests/test.sh

tests/build.sh

tests/test.sh

tests/build.sh

gemini-code-assist · 2025-08-07T15:30:11Z

tests/test.sh

+
+# This script runs disk image with qemu-system and run tmt against this vm.
+
+BOOTC_TEMPDIR="/tmp/tmp-bootc-build"


The temporary directory path /tmp/tmp-bootc-build is hardcoded here and in build.sh. This creates a tight, implicit coupling between the two scripts. It would be more robust to define this path in one place (e.g., in the Makefile), export it as an environment variable, and have both scripts use it. This would also make it easier to manage the lifecycle of the temporary directory (creation and cleanup).

Yes, I think we could just require this script to be invoked in the context of the temporary directory.

That said also, the only thing that actually binds the two things together right now is the ssh key.

But we can avoid that by not using --root-ssh-authorized-keys at bootc install to-disk time but instead injecting the SSH key via systemd credentials - that's how podman-bootc does it.

(Of course this whole topic instantly gets into the whole https://gitlab.com/fedora/bootc/tracker/-/issues/2 and whether/how podman-bootc and other virt provisioning tools should be shared underneath different testing frameworks)

Anyways for now though let's just change the GHA to allocate a temporary directory, see below

tmt/plans/bootc-integration.fmf

henrywang · 2025-08-08T14:50:11Z

@cgwalters , The two failures still need investigate. It's tmt issue, not our code issue.
Do you have any comment on this PR? Thanks.

cgwalters · 2025-08-08T15:22:31Z

.github/workflows/integration.yml

+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Install podman for heredoc support


Unrelated to this PR specifically but we should probably factor this stuff out into a shared action helper

cgwalters · 2025-08-08T15:24:23Z

.github/workflows/integration.yml

+  build:
+    strategy:
+      matrix:
+        test_os: [fedora-42]


Let's make this be quay.io/fedora/fedora:42 e.g. - see below

This test_os will be [fedora-41, fedora-42, fedora-43, centos-9, centos-10].

tests/build.sh

cgwalters · 2025-08-08T15:29:39Z

tests/test.sh

+
+# This script runs disk image with qemu-system and run tmt against this vm.
+
+BOOTC_TEMPDIR="/tmp/tmp-bootc-build"


Yes, I think we could just require this script to be invoked in the context of the temporary directory.

That said also, the only thing that actually binds the two things together right now is the ssh key.

But we can avoid that by not using --root-ssh-authorized-keys at bootc install to-disk time but instead injecting the SSH key via systemd credentials - that's how podman-bootc does it.

(Of course this whole topic instantly gets into the whole https://gitlab.com/fedora/bootc/tracker/-/issues/2 and whether/how podman-bootc and other virt provisioning tools should be shared underneath different testing frameworks)

Anyways for now though let's just change the GHA to allocate a temporary directory, see below

cgwalters · 2025-08-08T15:30:19Z

Makefile

 	cargo xtask test-tmt

+test:
+	tests/build.sh && tests/test.sh


I think this would then become:

tmpd=$(mktemp -d) && cd $tmpd && $srcdir/tests/build.sh && $srcdir/tests/test.sh

or so

cgwalters

I'm OK with this as is if you prefer, we can land further cleanups as followups

tests/build.sh

cgwalters · 2025-08-11T13:57:54Z

     package: building container image with dependencies

We had a live chat about this and I think the fact that tmt is dynamically rebuilding the OS image on boot is a problem. In the long term ideally the testing framework doesn't require installing any binaries at all on the target (this would let us test minimal systems without python etc) but:

In the short term how about tmt print-target-dependencies or so and then we can pre-install those in our container image, and then tmt shouldn't require rebuilding inside the target system? i.e. in the Dockerfile we do RUN dnf -y install $(tmt print-target-dependencies)

Signed-off-by: Xiaofeng Wang <[email protected]>

henrywang · 2025-09-10T13:33:49Z

tmt-1.56 fixed the blocker issue. @cgwalters I think we can land this PR. And I'll add https://github.com/cgwalters/bootc-kit into CI later in another PR. Thanks.

cgwalters

Thanks for working on this! Nothing truly blocking here so if you want to just merge as is go ahead and as always there's followups.

cgwalters · 2025-09-10T15:06:58Z

tests/test.sh

+ARCH=$(uname -m)
+case "$ARCH" in
+"aarch64")
+  qemu-system-aarch64 \


Eventually this is going to need to use a shared library of some form. Can't we just point tmt at the disk image and let testcloud run it or so?

tmt will call libvirt, not qemu. bootc-kit can replace those part of code later. I'm working on it now.

Yep and what I'm hitting now is that these qemu instances just leak after the test script runs, so trying to run multiple tests locally just fails.

I fixed this with a quick strategic use of setpriv --deathsig SIGTERM around qemu.

cgwalters · 2025-09-10T15:11:20Z

.github/workflows/integration.yml

+    needs: build
+    strategy:
+      matrix:
+        test_os: [fedora-41, fedora-42, fedora-43, centos-9]


This one is required to get different base image.

cgwalters · 2025-09-10T15:11:41Z

.github/workflows/integration.yml

+  build:
+    strategy:
+      matrix:
+        test_os: [fedora-41, fedora-42, fedora-43, centos-9]


I think we can stop testing fedora-41 overall and we MUST test centos-10

cgwalters · 2025-09-10T15:13:25Z

tests/build.sh

+# Base image
+case "$OS_ID" in
+    "centos")
+        TIER1_IMAGE_URL="quay.io/centos-bootc/centos-bootc:stream${OS_VERSION_ID}"


Not new and not directly related to this but we have this legacy of "TIER1" naming - there's only one default image and it's likely to stay that way. I'd probably just say BASE_IMAGE_URL going forward for things like this.

Right. Need update the name.

cgwalters · 2025-09-10T15:15:21Z

.github/workflows/integration.yml

+    strategy:
+      matrix:
+        test_os: [fedora-41, fedora-42, fedora-43, centos-9]
+        tmt_plan: [test-01-readonly, test-20-local-upgrade, test-21-logically-bound-switch, test-22-logically-bound-install, test-23-install-outside-container, test-24-local-upgrade-reboot]


It's going to be a maintenance trap if we have to list all tests here and in the tmt directory.

One thing that also seems like it will become a bit painful is having a CI context check per test. (It's not wrong, but it will get really long)

A good middle ground might be "sharding"; basically like https://nexte.st/docs/ci-features/partitioning/?h=shard#hashed-sharding where we run a group of tests per GHA runner. That would give us a good mix of parallelism and efficiency.

And crucially by doing the sharding we would avoid needing to list each test; the allocated runner discovers which tests it should run by just taking an integer from 0-N as input, and looking if the hash of the test modulo the input key is zero.

cgwalters · 2025-09-10T15:17:23Z

tests/build.sh

+        ;;
+esac
+
+CONTAINERFILE="${BOOTC_TEMPDIR}/Containerfile"


This would create a third copy of our Dockerfile, I think we can avoid that by just having the github action matrix be directly passed down into e.g.

base=$1 shift podman build --build-arg=base=$base

And that's all right?

cgwalters · 2025-09-10T15:20:15Z

.github/workflows/integration.yml

@@ -0,0 +1,100 @@
+name: bootc integration test


Biggest question: Will we also continue to run the tests via packit+testing farm? Probably not, right?

But still though, we should bear in mind that we still need to have good bridges/integration with at least tmt/testing-farm since other groups will use that.

No, test can be run on github action runner and with make test locally.

I thought more about this and I think there's an obvious thing to keep doing with the packit+testing-farm flow, which is testing of system-reinstall-bootc.

For that we very precisely want an RPM and a stock cloud image. It's actually almost a good thing if we're not testing having a new bootc in the target container image just to shake out any cross-version issues.

cgwalters · 2025-09-11T17:06:23Z

OK let's get this in and then continue iteration!

gemini-code-assist bot reviewed Aug 7, 2025

View reviewed changes

henrywang force-pushed the qemu_tmt branch from cfcb683 to d9d7182 Compare August 7, 2025 15:31

henrywang changed the title ~~test: Add integration test running on github runner~~ [WIP] test: Add integration test running on github runner Aug 7, 2025

henrywang force-pushed the qemu_tmt branch 7 times, most recently from 1ce535b to a2def9a Compare August 8, 2025 07:13

cgwalters reviewed Aug 8, 2025

View reviewed changes

henrywang force-pushed the qemu_tmt branch 4 times, most recently from 33c2934 to 3067f33 Compare August 11, 2025 10:28

cgwalters approved these changes Aug 11, 2025

View reviewed changes

tests/build.sh Outdated Show resolved Hide resolved

henrywang force-pushed the qemu_tmt branch 4 times, most recently from eac749f to 07ef754 Compare August 12, 2025 12:01

cgwalters mentioned this pull request Aug 12, 2025

Create installation VM and run bootc install inside a VM using rootless podman bootc-dev/podman-bootc#95

Draft

henrywang force-pushed the qemu_tmt branch 6 times, most recently from 9347e32 to 9f5e783 Compare August 13, 2025 09:45

test: Add integration test running on github runner

54f8562

Signed-off-by: Xiaofeng Wang <[email protected]>

henrywang added 3 commits September 10, 2025 20:55

test: Run script with sudo instead of adding sudo in script

cc53ba1

Signed-off-by: Xiaofeng Wang <[email protected]>

ci: Add bootc and image build on ARM

310bc53

Signed-off-by: Xiaofeng Wang <[email protected]>

ci: Add fedora 41/42/43 and cs9/10 integration test

49cca3a

Signed-off-by: Xiaofeng Wang <[email protected]>

henrywang force-pushed the qemu_tmt branch from 9f5e783 to 49cca3a Compare September 10, 2025 12:55

henrywang changed the title ~~[WIP] test: Add integration test running on github runner~~ test: Add integration test running on github runner Sep 10, 2025

cgwalters approved these changes Sep 10, 2025

View reviewed changes

cgwalters merged commit 029ed34 into bootc-dev:main Sep 11, 2025
60 checks passed

henrywang deleted the qemu_tmt branch September 24, 2025 07:16


		# This script runs disk image with qemu-system and run tmt against this vm.

		BOOTC_TEMPDIR="/tmp/tmp-bootc-build"

test: Add integration test running on github runner #1496

test: Add integration test running on github runner #1496

Uh oh!

Conversation

henrywang commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Aug 7, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

henrywang commented Aug 8, 2025

Uh oh!

cgwalters Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cgwalters commented Aug 11, 2025

Uh oh!

henrywang commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

henrywang Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

henrywang commented Aug 7, 2025 •

edited

Loading

cgwalters Aug 8, 2025 •

edited

Loading

henrywang commented Sep 10, 2025 •

edited

Loading

henrywang Sep 11, 2025 •

edited

Loading