grub-boot-success.timer interfering with greenboot

[say-paul]

ssh session seems to trigger the grub-boot-success.timer from grub2-tools to set the boot success flag, which means that grub does not decrement the boot counter if there is a reboot.

[nullr0ute]

Going to need more concrete evidence that a brief statement

[pmtk]

I encountered this issue recently, I can provide some details.

Package grub2-tools provides two files involved in this problem:

/usr/lib/systemd/user/grub-boot-success.service
/usr/lib/systemd/user/grub-boot-success.timer

After user logs in (e.g. with SSH), the timer starts and after 2 minutes the service runs (ExecStart=/usr/sbin/grub2-set-bootflag boot_success).

The same package provides /etc/grub.d/08_fallback_counting in which we can see:

# Check if boot_counter exists and boot_success=0 to activate this behaviour.
if [ -n "\${boot_counter}" -a "\${boot_success}" = "0" ]; then

It means that: if user logs in during the healthchecks and running healthchecks takes longer than these 2 minutes needed to trigger grub-boot-success.service, then grub won't decrement the boot_counter resulting in more "reboots to heal" than expected (e.g configured in greenboot).

This is a log I obtained when discovered the issue:

$ sudo journalctl -u greenboot-healthcheck | grep -E "\-\- Boot|boot_[s|c]"
-- Boot c3172dea78584d9b9879f818be8c1f34 --
Jul 05 12:36:41 ostree-dev 40_microshift_running_check.sh[1189]: boot_success=0
Jul 05 12:36:41 ostree-dev 40_microshift_running_check.sh[1189]: boot_counter=2

-- Boot 4081b57ed6284b07a48c105c5be9a79e --
Jul 05 12:41:12 ostree-dev 40_microshift_running_check.sh[1233]: boot_success=0
Jul 05 12:41:12 ostree-dev 40_microshift_running_check.sh[1233]: boot_counter=2

-- Boot e7fc9bb7e24f4ea080c7c4d486533567 --
Jul 05 12:44:45 ostree-dev 40_microshift_running_check.sh[1281]: boot_success=0
Jul 05 12:44:45 ostree-dev 40_microshift_running_check.sh[1281]: boot_counter=2

-- Boot 9e77bc317ad54761845d53118ef52c9b --
Jul 05 12:48:21 ostree-dev 40_microshift_running_check.sh[1222]: boot_success=0
Jul 05 12:48:21 ostree-dev 40_microshift_running_check.sh[1222]: boot_counter=2

-- Boot 9db0cd6a1bd74bb6a2ef3bfb19bfc2e5 --
Jul 05 12:51:58 ostree-dev 40_microshift_running_check.sh[1190]: boot_success=0
Jul 05 12:51:58 ostree-dev 40_microshift_running_check.sh[1190]: boot_counter=1

-- Boot 05f39c32ef48400ea07cb000af463ecb --
Jul 05 12:55:16 ostree-dev 40_microshift_running_check.sh[1283]: boot_success=0
Jul 05 12:55:16 ostree-dev 40_microshift_running_check.sh[1283]: boot_counter=1

-- Boot bbac60cff9a84ef19ed99a949dd38d97 --
Jul 05 12:58:50 ostree-dev 40_microshift_running_check.sh[1189]: boot_success=0
Jul 05 12:58:50 ostree-dev 40_microshift_running_check.sh[1189]: boot_counter=1

-- Boot b2c67e9a44764388aa4c9fd120141a2c --
Jul 05 13:02:30 ostree-dev 40_microshift_running_check.sh[1187]: boot_success=0
Jul 05 13:02:30 ostree-dev 40_microshift_running_check.sh[1187]: boot_counter=0

-- Boot d2e24133fd294f5580f40cd5537512ed --

[LorbusChris]

To me it looks like grub-boot-success.{service,timer} should be excluded from Fedora IoT and other OSes that use greenboot.

[LorbusChris]

Further, if grub-boot-success.service and grub2-set-success.service were more aligned (i.e. grub-boot-success.service would also ensure the boot_counter is unset) Fedora IoT could also just drop the timer, drop greenboot-grub2-set-success.service and run with the grub-boot-success.service.

[say-paul]

We can use this as a workaround:
systemctl --user mask grub-boot-success.timer

[say-paul]

@LorbusChris I there a way to drop just these two: grub-boot-success.{service,timer}
or we may need to remove the entire rpm: grub2-tools , If its the later then I think we can go with masking them unless we understand the bigger implications of removing the entire repo.

[LorbusChris]

IMO This should be solved by using preset that ensures grub-boot-success.{service,timer} is disabled on Fedora IoT/RHEL Edge. They should really only be enabled on Desktop variants, not on Server/IoT.

[LorbusChris]

Less clean and less preferable, but also working as you mentioned, would be to mask them, which you might be able to do in osbuild/Image Builder.

[pmtk]

We use masking when testing MicroShift.

I'm not sure, but it looks like osbuild might not support masking, only enabling / disabling (which does not work for timers): https://www.osbuild.org/guides/image-builder-on-premises/blueprint-reference.html#systemd-services

Unless you have another idea how to handle it? We were also thinking about putting systemctl [--user] mask in the kickstart

[LorbusChris]

On Fedora IoT, this timer is already be disabled by this preset: https://src.fedoraproject.org/rpms/fedora-release/blob/rawhide/f/80-iot-user.preset#_6

This is included in the fedora-release-iot rpm.

Doing this via a preset config is the proper way to do it.

[pmtk]

But this is method for Fedora IoT only, right?
For RHEL For Edge documentation recommends osbuild and I'm not aware of such option in the blueprint. Or is there a way for R4E?

[say-paul]

Evaluating ways to do in osbuild-composer also, might require some modifications in the source code.

[say-paul]

The fix PR can be tracked here : osbuild/images#51