kola: ext.config.rpm-ostree.kernel-replace fails with "no space left on device"

[nikita-dubrovskii]

Test works locally, but fails on builder(s) (at least x86):

[2025-07-16T15:50:59.713Z] Jul 16 15:50:48 qemu0 kola-runext-kernel-replace[2384]: time="2025-07-16T15:50:48Z" level=error msg="Can't add file /var/tmp/container_images_oci3147168388/blobs/sha256/874add5eed3680ce6f0739014d098ac4307476d49ae7fa5679e8ba8003e05ab0 to tar: io: read/write on closed pipe"
[2025-07-16T15:50:59.713Z] Jul 16 15:50:49 qemu0 podman[2384]: 2025-07-16 15:50:49.219862937 +0000 UTC m=+193.004311406 image build  
[2025-07-16T15:50:59.713Z] Jul 16 15:50:49 qemu0 kola-runext-kernel-replace[2384]: Error: committing container for step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[rpm-ostree override replace /tmp/buildcontext/*rpm &&     rpm-ostree cleanup -m &&     ostree container commit] Flags:[] Attrs:map[] Message:RUN rpm-ostree override replace /tmp/buildcontext/*rpm &&     rpm-ostree cleanup -m &&     ostree container commit Heredocs:[] Original:RUN rpm-ostree override replace /tmp/buildcontext/*rpm &&     rpm-ostree cleanup -m &&     ostree container commit}: copying layers and metadata for container "bd91ca4f0d8ccdcd16ac684b5ec9a7eca0c94cf4803d2a73e1c868388e64f1f3": committing the finished image: write /var/tmp/coreos-derived.ociarchive: no space left on device
[2025-07-16T15:50:59.713Z] Jul 16 15:50:49 qemu0 systemd[1]: kola-runext.service: Main process exited, code=exited, status=125/n/a
[2025-07-16T15:50:59.713Z] Jul 16 15:50:49 qemu0 systemd[1]: kola-runext.service: Failed with result 'exit-code'.
[2025-07-16T15:50:59.713Z] Jul 16 15:50:49 qemu0 systemd[1]: kola-runext.service: Consumed 1min 28.189s CPU time, 1.5G memory peak.
[2025-07-16T15:50:59.713Z] --- �[31mFAIL�[0m: ext.config.rpm-ostree.kernel-replace (270.84s)
[2025-07-16T15:50:59.713Z]         harness.go:1309: kolet failed: : kolet run-test-unit failed:  Error: Unit kola-runext.service exited with code 125

[dustymabe]

Where are you hitting this?

We hit something similar in zincati CI, but changed to using a non debug rust build for the test as a workaround.

We should be able to change the test to specify MinDisk to bump it up from the default if we find it's really necessary.

[jbtrystram]

Where are you hitting this?

I think this was happening consistently in the fcos pipeline this week

[jlebon]

Where are you hitting this?

I think this was happening consistently in the fcos pipeline this week

Are you sure it was in the production pipeline and not CI? Where I've seen it recently is:

• as Dusty mentioned in Zincati, which was fixed by Cargo.toml: stop setting debug=true for release profile; rework CI zincati#1306
• In CI for Add cosa import coreos-assembler#4164, which was fixed by Containerfile: run postprocess scripts in build stage fedora-coreos-config#3598

It definitely may be that we need to bump requirements, but I'd also be interested in knowing where exactly this was hit in case it actually signals something else.

[jlebon]

OK, possibly Nikita filed this from seeing this in CI in coreos/coreos-installer#1677, which I just noticed had failed on that too.

We do strip rdcore there, but not coreos-installer. Possibly we should strip that too or do something similar to what we did in Zincati of building the release build in CI and turning off debug symbols.

But TBH, also fine with bumping the test requirements.

[dustymabe]

Let's see if CI passes on coreos/coreos-installer#1682 and consider merging it if so.

[nikita-dubrovskii]

@jlebon

0K, possibly Nikita filed this from seeing this in CI in coreos/coreos-installer#1677, which I just noticed had failed on that too.

Yep, exactly.

@dustymabe

Let's see if CI passes on coreos/coreos-installer#1682 and consider merging it if so.

Thank you! I guess we can close this issue now