Watch the size of /boot when running updates

[RagnarSTS]

I just had a sensor node that had a /boot of 500M that was too full to do upgrades.
IIRC the system started on 2.4.10, upgraded to 2.4.20, and failed temporarily bumping to 2.4.30.
I think it had a pair of 5.14 and 5.15 kernels and lacked a bit of space for a 5th. Just need to keep things tidy in the /boot.

[dougburks]

Was this sensor installed from our Security Onion ISO image or did you perform a network installation?

[RagnarSTS]

It was seco 2.4.10-20230821.iso

[RagnarSTS]

No extra packages installed along the way; but you do seem to have confirmed that there are 4 kernels sitting there not 2.
Your 2.4.5RC added a third kernel and 2.4.20 added a fourth.

Bumping from 2.4.20 > 2.4.30 did this?:

kernel-uek-5.15.0-103.114.4.el9uek.x86_64      >>  (removed)
kernel-uek-5.15.0-105.125.6.2.2.el9uek.x86_64  >>  kernel-uek-5.15.0-105.125.6.2.2.el9uek.x86_64 (nochange)
                                               >>  kernel-uek-5.15.0-200.131.27.el9uek.x86_64 (added)
kernel-5.14.0-284.25.1.0.1.el9_2.x86_64        >>  (removed)
kernel-5.14.0-284.30.0.1.el9_2.x86_64          >>  kernel-5.14.0-284.30.0.1.el9_2.x86_64 (nochange)
                                               >>  kernel-5.14.0-362.8.1.el9_3.x86_64 (added)

Guessing the process was add two updated kernels and only after successful completion remove the two older ones...might have been pinched there temporarily for some reason.

Even my lab setup it only failed out on one node, and didn't seem 100% full in the first place. 97M free on your example is indeed a little tight, but wouldn't normally cause a failure.
Frequently as dedicated mount points /boot recommended minimum size is 1G and /boot/EFI would be 200M or so. We're kind of splitting the difference in the middle at 500 and just need to be slightly more aware of limitations.

[dougburks]

you do seem to have confirmed that there are 4 kernels sitting there not 2

The installonly_limit=2 setting allows 2 versions of the standard RedHat kernel (the standard kernel package) and 2 versions of the Oracle kernel (the kernel-uek package).

Bumping from 2.4.20 > 2.4.30 did this?:

Yes, it appears that way.

Guessing the process was add two updated kernels and only after successful completion remove the two older ones...might have been pinched there temporarily for some reason.
Even my lab setup it only failed out on one node, and didn't seem 100% full in the first place. 97M free on your example is indeed a little tight, but wouldn't normally cause a failure.
Frequently as dedicated mount points /boot recommended minimum size is 1G and /boot/EFI would be 200M or so. We're kind of splitting the difference in the middle at 500 and just need to be slightly more aware of limitations.

I repeated my test above but this time with the VM set to EFI instead of BIOS and it worked fine without issues again. Full data below and it's almost identical to the first test run.

After installing 2.4.4 RC1:

[root@securityonion ~]# cat /etc/soversion
2.4.4
[root@securityonion ~]# rpm -qa |egrep "kernel-5|kernel-uek-5"
kernel-5.14.0-284.18.1.el9_2.x86_64
kernel-uek-5.15.0-103.114.4.el9uek.x86_64
[root@securityonion ~]# df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       507M  272M  236M  54% /boot

After soup to 2.4.5:

[root@securityonion ~]# cat /etc/soversion 
2.4.5
[root@securityonion ~]# rpm -qa |egrep "kernel-5|kernel-uek-5"
kernel-5.14.0-284.18.1.el9_2.x86_64
kernel-uek-5.15.0-103.114.4.el9uek.x86_64
kernel-5.14.0-284.25.1.el9_2.x86_64
[root@securityonion ~]# df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       507M  322M  185M  64% /boot

After soup to 2.4.10:

[root@securityonion ~]# cat /etc/soversion 
2.4.10
[root@securityonion ~]# rpm -qa |egrep "kernel-5|kernel-uek-5"
kernel-uek-5.15.0-103.114.4.el9uek.x86_64
kernel-5.14.0-284.25.1.el9_2.x86_64
kernel-5.14.0-284.25.1.0.1.el9_2.x86_64
[root@securityonion ~]# df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       507M  318M  189M  63% /boot

After soup to 2.4.20:

[root@securityonion ~]# cat /etc/soversion 
2.4.20
[root@securityonion ~]# rpm -qa |egrep "kernel-5|kernel-uek-5"
kernel-uek-5.15.0-103.114.4.el9uek.x86_64
kernel-5.14.0-284.25.1.0.1.el9_2.x86_64
kernel-uek-5.15.0-105.125.6.2.2.el9uek.x86_64
kernel-5.14.0-284.30.0.1.el9_2.x86_64
[root@securityonion ~]# df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       507M  403M  105M  80% /boot

After soup to 2.4.30:

[root@securityonion ~]# cat /etc/soversion 
2.4.30
[root@securityonion ~]# rpm -qa |egrep "kernel-5|kernel-uek-5"
kernel-uek-5.15.0-105.125.6.2.2.el9uek.x86_64
kernel-5.14.0-284.30.0.1.el9_2.x86_64
kernel-5.14.0-362.8.1.el9_3.x86_64
kernel-uek-5.15.0-200.131.27.el9uek.x86_64
[root@securityonion ~]# df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       507M  405M  103M  80% /boot

[RagnarSTS]

Computers are silly.

[dougburks]

Computers are silly.

Yes, teaching sand to think was a mistake!

In any case, we'll continue to keep an eye on this issue and see if anybody else runs into it.

[Acewiza]

It's like kids - Wadda ya expect given 'em dangerous toys like heat and light.

[RagnarSTS]

Perhaps use /boot/efi mountpoint for the 500M partition and keep the rest of /boot under normal / mountpoint.

[Acewiza]

I like the way the script tells you exactly what's wrong. Bigger /boot maybe better, if you like keeping stuff hanging around. Probably extrapolate a whole major version of updates based on x#fixes/month for a couple years without much of a disk premium these days.

[Sazerak]

I just came across this thread as I ran into the same issue. Tried running soup which failed due to lack of space on /boot. I did the checks suggested above but it only confirmed that there was in fact a lack of space. I ended up running dnf remove on the packages that were to be removed by dnf update and that seems to have fixed things (update is still going as I'm writing this, but seems to have made it farther than last try). This was on a standalone ISO install and the install limit was at 2.

[dougburks]

@Sazerak Just to confirm, are you using our Security Onion ISO image?

What version did you originally install?

What versions have you upgraded to since then?

Have you installed any other software?

Have you added any other files or directories to /boot?

As I mentioned above, I've tested this multiple times but have not been able to duplicate it. Are you able to provide steps to duplicate?

[Sazerak]

Hi Doug, to be honest SO2 has taken a bit of a back seat for me recently between family and work obligations, so I can't recall much of when/how I moved from 2.3 to 2.4, but what I can tell you is that it was a fresh ISO install which I think I started at 2.4.10. I probably upgraded to 2.4.20 before upgrading again today, but I can't say for sure. Nothing else has been installed and I'm not aware of anything else in /boot that shouldn't be there (nothing stood out when I checked).

Seeing as the upgrade worked now that I ran dnf remove as mentioned above, I don't think I'd be able to reproduce the steps, but if there are any logs or whatnot you'd like me to post here that you think might help, please don't hesitate to ask. I'm always happy to help! Unfortunately I didn't take screenshots, but I was lacking 19M in /boot which made the install fail. When I check with sudo dnf update I got the same results. Once I removed the excess kernels, the update went smoothly.

[dougburks]

Starting in 2.4.40, our ISO image will set /boot to 1GB:
#12002

[Acewiza]

I just whacked the boot stuff I didn't want any longer. ;-)