Test flake in test-20-image-pushpull-upgrade

[jeckersb]

From https://github.com/bootc-dev/bootc/actions/runs/19977938612/job/57438955783?pr=1832

content: Adding quoted karg via rpm-ostree to test ostree issue #3544
content: error: System transaction in progress

Not immediately clear to me if this is fighting with the bootc switch immediately before for the lock, or if something else is running in the tmt environment that's grabbing it.

[cgwalters]

Yeah this one is just ruining our CI suddenly. I'll investigate

[jeckersb]

A few more explicit datapoints:

• It seems to only be happening on the centos10 job
• It seems to be reliably reproducible (we have retried the job many times now and it keeps failing consistently)
• If I run just test-tmt image-pushpull-upgrade (which defaults to centos10) locally on my laptop, it works fine (sample size of one)

[cgwalters]

I can make rpm-ostree emit this error by spawning a few tight loops with a process which grabs the sysroot lock like while true; do ostree admin pin 0 &>/dev/null; done & for example.

The rpm-ostree code seems to very explicitly not be attempting to block here, and I think has always been subject to this problem. The question though is what other process is grabbing the lock here. It might be bootc-fetch-apply-updates.timer - which yeah, we probably need to consistently disable.

Still investigating.

[jeckersb]

FWIW I almost forgot that we discussed earlier rebasing #1832, so I just did that. We'll see if it makes any difference 👀

[henrywang]

I have a kind of workaround for this flaky in 4e1d6ad to make sure rpm-ostree not in System transaction in progress before run rpm-ostree kargs --append=($quoted_karg). The idea is simple, just run a simple rpm-ostree command first before run test to avoid flaky.

This seems work in recently CI result.

[jeckersb]

Yeah looks like rebasing #1832 to include @henrywang's fix made this go away. I'm fine closing this now but if we want to leave it open to dig a bit more into the root cause that's fine too.