Unable to install a sensor - mismatch in kernel modules

[rosswakelin]

Version

2.4.60

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

32

Storage for /

250Gb

Storage for /nsm

1Tb

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I have two grids, one using 2.3 and a new one using 2.4. On the 2.4 grid, the devices are all running 2.4.60, having been upgraded from 2.4.40.
I went to add a new sensor that was pre-staged with a non-configured 2.4.40 ISO install. We pre-stage the iso install, ship the device to site, then finish the install remotely. This has worked fine for us up to now but....
We go through the setup screens, entering all the details, the new sensor find the manager, then starts the install process. Part of this install process is downloading updates etc. from the manager. The install is failing downloading files (in particular kernel modules etc.) from the repo on the manager. The repo on the manager has kernel files for 5.14.0-362.24, but the install process is trying to find 5.14.0-427.13. Which of course, it can't find.
Is there an issue with doing an "upgrade install"? Why is the installer looking for a newer version than is in the repo?

Here is the install log...

Imported RPM-GPG-KEY-oracle
Imported RPM-GPG-KEY-EPEL-9
Imported SALT-PROJECT-GPG-PUBKEY-2023.pub
Imported docker.pub
Imported securityonion.pub
2024-05-07T04:55:15Z | INFO | Executing command: dnf -v clean all
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
DNF version: 4.14.0
cachedir: /var/cache/dnf
Cleaning data: dbcache metadata packages
5 files removed
2024-05-07T04:55:15Z | INFO | Executing command: mkdir -vp /root/oldrepos
2024-05-07T04:55:15Z | INFO | Executing command: mv -v /etc/yum.repos.d/* /root/oldrepos/
renamed '/etc/yum.repos.d/securityonion.repo' -> '/root/oldrepos/securityonion.repo'
2024-05-07T04:55:15Z | INFO | Executing command: dnf repolist all
repo id repo name status
securityonion Security Onion Repo enabled
2024-05-07T04:55:15Z | INFO | Executing command: dnf repolist
repo id repo name
securityonion Security Onion Repo
2024-05-07T04:55:16Z | INFO | Executing command: dnf -y update --allowerasing --exclude=salt*,docker*,containerd*
Security Onion Repo 3.3 MB/s | 3.5 MB 00:01
Last metadata expiration check: 0:00:01 ago on Tue 07 May 2024 04:55:17 AM UTC.
Dependencies resolved.

Package Arch Version Repository Size

Installing:
kernel x86_64 5.14.0-427.13.1.el9_4 securityonion 5.8 M
Upgrading:
NetworkManager x86_64 1:1.46.0-4.0.1.el9_4 securityonion 2.3 M
NetworkManager-libnm x86_64 1:1.46.0-4.0.1.el9_4 securityonion 1.9 M
NetworkManager-team x86_64 1:1.46.0-4.0.1.el9_4 securityonion 40 k
NetworkManager-tui x86_64 1:1.46.0-4.0.1.el9_4 securityonion 252 k
NetworkManager-wifi x86_64 1:1.46.0-4.0.1.el9_4 securityonion 84 k
acl x86_64 2.3.1-4.el9 securityonion 87 k
audit x86_64 3.1.2-2.el9 securityonion 330 k
audit-libs x86_64 3.1.2-2.el9 securityonion 122 k
avahi-libs x86_64 0.8-20.el9 securityonion 73 k
bash x86_64 5.1.8-9.el9 securityonion 1.8 M
bind-libs x86_64 32:9.16.23-15.0.1.el9 securityonion 1.2 M
bind-license noarch 32:9.16.23-15.0.1.el9 securityonion 11 k
bind-utils x86_64 32:9.16.23-15.0.1.el9 securityonion 224 k
binutils x86_64 2.35.2-43.0.1.el9 securityonion 4.8 M
binutils-gold x86_64 2.35.2-43.0.1.el9 securityonion 746 k
btrfs-progs x86_64 5.15.1-1.el9 securityonion 984 k
chrony x86_64 4.5-1.0.1.el9 securityonion 355 k
coreutils x86_64 8.32-35.0.1.el9 securityonion 1.2 M
coreutils-common x86_64 8.32-35.0.1.el9 securityonion 2.1 M
cronie x86_64 1.5.7-11.el9 securityonion 133 k
cronie-anacron x86_64 1.5.7-11.el9 securityonion 37 k
curl x86_64 7.76.1-29.el9_4 securityonion 304 k
device-mapper x86_64 9:1.02.197-2.el9 securityonion 156 k
device-mapper-event x86_64 9:1.02.197-2.el9 securityonion 38 k
device-mapper-event-libs x86_64 9:1.02.197-2.el9 securityonion 31 k
device-mapper-libs x86_64 9:1.02.197-2.el9 securityonion 177 k
device-mapper-persistent-data x86_64 1.0.9-3.el9_4 securityonion 1.0 M
dmidecode x86_64 1:3.5-3.el9 securityonion 108 k
dracut x86_64 057-53.git20240104.0.1.el9 securityonion 676 k
dracut-config-rescue x86_64 057-53.git20240104.0.1.el9 securityonion 13 k
dracut-network x86_64 057-53.git20240104.0.1.el9 securityonion 148 k
dracut-squash x86_64 057-53.git20240104.0.1.el9 securityonion 13 k
e2fsprogs x86_64 1.46.5-5.el9 securityonion 1.1 M
e2fsprogs-libs x86_64 1.46.5-5.el9 securityonion 221 k
elfutils-debuginfod-client x86_64 0.190-2.el9 securityonion 42 k
elfutils-libelf x86_64 0.190-2.el9 securityonion 202 k
elfutils-libs x86_64 0.190-2.el9 securityonion 258 k
expat x86_64 2.5.0-2.el9_4 securityonion 122 k
file x86_64 5.39-16.el9 securityonion 55 k
file-libs x86_64 5.39-16.el9 securityonion 596 k
fuse-common x86_64 3.10.2-8.el9 securityonion 7.2 k
git x86_64 2.43.0-1.el9 securityonion 65 k
git-core x86_64 2.43.0-1.el9 securityonion 4.5 M
git-core-doc noarch 2.43.0-1.el9 securityonion 3.9 M
glib2 x86_64 2.68.4-14.el9 securityonion 2.7 M
glibc x86_64 2.34-100.0.1.el9 securityonion 2.0 M
glibc-common x86_64 2.34-100.0.1.el9 securityonion 331 k
glibc-gconv-extra x86_64 2.34-100.0.1.el9 securityonion 1.8 M
glibc-langpack-en x86_64 2.34-100.0.1.el9 securityonion 969 k
gnutls x86_64 3.8.3-1.el9 securityonion 1.1 M
grub2-common noarch 1:2.06-77.0.1.el9 securityonion 955 k
grub2-pc x86_64 1:2.06-77.0.1.el9 securityonion 14 k
grub2-pc-modules noarch 1:2.06-77.0.1.el9 securityonion 1.2 M
grub2-tools x86_64 1:2.06-77.0.1.el9 securityonion 1.9 M
grub2-tools-minimal x86_64 1:2.06-77.0.1.el9 securityonion 620 k
grubby x86_64 8.40-64.0.1.el9 securityonion 43 k
httpd-tools x86_64 2.4.57-8.0.1.el9 securityonion 95 k
initscripts x86_64 10.11.6-1.el9 securityonion 285 k
initscripts-rename-device x86_64 10.11.6-1.el9 securityonion 14 k
iproute x86_64 6.2.0-6.el9_4 securityonion 894 k
iptables-libs x86_64 1.8.10-2.el9 securityonion 506 k
iptables-nft x86_64 1.8.10-2.el9 securityonion 218 k
iw x86_64 6.7-1.el9 securityonion 120 k
iwl100-firmware noarch 999:39.31.5.1-999.31.el9 securityonion 126 k
iwl1000-firmware noarch 999:39.31.5.1-999.31.el9 securityonion 126 k
iwl105-firmware noarch 999:18.168.6.1-999.31.el9 securityonion 205 k
iwl135-firmware noarch 999:18.168.6.1-999.31.el9 securityonion 213 k
iwl2000-firmware noarch 999:18.168.6.1-999.31.el9 securityonion 207 k
iwl2030-firmware noarch 999:18.168.6.1-999.31.el9 securityonion 216 k
iwl3160-firmware noarch 999:25.30.13.0-999.31.el9 securityonion 482 k
iwl5000-firmware noarch 999:8.83.5.1_1-999.31.el9 securityonion 123 k
iwl5150-firmware noarch 999:8.24.2.2-999.31.el9 securityonion 123 k
iwl6000g2a-firmware noarch 999:18.168.6.1-999.31.el9 securityonion 191 k
iwl6000g2b-firmware noarch 999:18.168.6.1-999.31.el9 securityonion 193 k
iwl6050-firmware noarch 999:41.28.5.1-999.31.el9 securityonion 148 k
iwl7260-firmware noarch 999:25.30.13.0-999.31.el9 securityonion 6.2 M
iwlax2xx-firmware noarch 999:20240122-999.31.el9 securityonion 77 M
jq x86_64 1.6-16.el9 securityonion 194 k
kernel-tools x86_64 5.14.0-427.13.1.el9_4 securityonion 6.1 M
kernel-tools-libs x86_64 5.14.0-427.13.1.el9_4 securityonion 5.8 M
kexec-tools x86_64 2.0.28-1.0.4.el9 securityonion 539 k
kpartx x86_64 0.8.7-27.el9 securityonion 56 k
less x86_64 590-3.el9_3 securityonion 175 k
libacl x86_64 2.3.1-4.el9 securityonion 22 k
libblkid x86_64 2.37.4-18.0.1.el9 securityonion 107 k
libbpf x86_64 2:1.3.0-2.el9 securityonion 172 k
libcom_err x86_64 1.46.5-5.el9 securityonion 25 k
libcurl x86_64 7.76.1-29.el9_4 securityonion 283 k
libdnf x86_64 0.69.0-8.0.1.el9 securityonion 709 k
libdrm x86_64 2.4.117-1.el9 securityonion 165 k
libertas-sd8787-firmware noarch 999:20240122-999.31.gitbf0987d3.el9 securityonion 299 k
libfdisk x86_64 2.37.4-18.0.1.el9 securityonion 154 k
libfido2 x86_64 1.13.0-2.el9 securityonion 102 k
libgcc x86_64 11.4.1-3.0.1.el9 securityonion 98 k
libgomp x86_64 11.4.1-3.0.1.el9 securityonion 271 k
libibverbs x86_64 48.0-1.el9 securityonion 419 k
libkcapi x86_64 1.4.0-2.0.1.el9 securityonion 53 k
libkcapi-hmaccalc x86_64 1.4.0-2.0.1.el9 securityonion 36 k
libmount x86_64 2.37.4-18.0.1.el9 securityonion 134 k
libnftnl x86_64 1.2.6-2.el9 securityonion 82 k
libnl3 x86_64 3.9.0-1.el9 securityonion 361 k
libnl3-cli x86_64 3.9.0-1.el9 securityonion 276 k
librepo x86_64 1.14.5-2.el9 securityonion 87 k
libselinux x86_64 3.6-1.el9 securityonion 85 k
libselinux-utils x86_64 3.6-1.el9 securityonion 257 k
libsemanage x86_64 3.6-1.el9 securityonion 122 k
libsepol x86_64 3.6-1.el9 securityonion 329 k
libsmartcols x86_64 2.37.4-18.0.1.el9 securityonion 63 k
libss x86_64 1.46.5-5.el9 securityonion 31 k
libssh x86_64 0.10.4-13.el9 securityonion 222 k
libssh-config noarch 0.10.4-13.el9 securityonion 9.4 k
libstdc++ x86_64 11.4.1-3.0.1.el9 securityonion 754 k
libtirpc x86_64 1.3.3-8.el9_4 securityonion 100 k
libuuid x86_64 2.37.4-18.0.1.el9 securityonion 28 k
linux-firmware noarch 999:20240122-999.31.gitbf0987d3.el9 securityonion 375 M
linux-firmware-core noarch 999:20240122-999.31.gitbf0987d3.el9 securityonion 746 k
linux-firmware-whence noarch 999:20240122-999.31.gitbf0987d3.el9 securityonion 42 k
lvm2 x86_64 9:2.03.23-2.el9 securityonion 1.6 M
lvm2-libs x86_64 9:2.03.23-2.el9 securityonion 1.0 M
mdadm x86_64 4.2-12.el9_4 securityonion 461 k
nettle x86_64 3.9.1-1.el9 securityonion 569 k
nftables x86_64 1:1.0.9-1.el9 securityonion 437 k
numactl-libs x86_64 2.0.16-3.el9 securityonion 29 k
open-vm-tools x86_64 12.3.5-2.0.1.el9 securityonion 964 k
openldap x86_64 2.6.6-3.el9 securityonion 277 k
openssh x86_64 8.7p1-38.0.1.el9 securityonion 484 k
openssh-clients x86_64 8.7p1-38.0.1.el9 securityonion 735 k
openssh-server x86_64 8.7p1-38.0.1.el9 securityonion 477 k
openssl x86_64 1:3.0.7-27.0.3.el9 securityonion 1.3 M
openssl-libs x86_64 1:3.0.7-27.0.3.el9 securityonion 1.9 M
oraclelinux-release x86_64 9:9.4-1.0.6.el9 securityonion 8.3 M
p11-kit x86_64 0.25.3-2.el9 securityonion 599 k
p11-kit-trust x86_64 0.25.3-2.el9 securityonion 147 k
pam x86_64 1.5.1-19.0.1.el9 securityonion 786 k
pcp-conf x86_64 6.2.0-1.0.1.el9 securityonion 36 k
pcp-libs x86_64 6.2.0-1.0.1.el9 securityonion 640 k
pcre2 x86_64 10.40-5.0.1.el9 securityonion 234 k
pcre2-syntax noarch 10.40-5.0.1.el9 securityonion 155 k
perl-B x86_64 1.80-481.el9 securityonion 195 k
perl-DynaLoader x86_64 1.47-481.el9 securityonion 24 k
perl-Errno x86_64 1.30-481.el9 securityonion 13 k
perl-Fcntl x86_64 1.13-481.el9 securityonion 19 k
perl-Git noarch 2.43.0-1.el9 securityonion 42 k
perl-IO x86_64 1.43-481.el9 securityonion 109 k
perl-NDBM_File x86_64 1.15-481.el9 securityonion 21 k
perl-POSIX x86_64 1.94-481.el9 securityonion 99 k
perl-interpreter x86_64 4:5.32.1-481.el9 securityonion 76 k
perl-lib x86_64 0.65-481.el9 securityonion 13 k
perl-libs x86_64 4:5.32.1-481.el9 securityonion 2.7 M
perl-mro x86_64 1.23-481.el9 securityonion 27 k
policycoreutils x86_64 3.6-2.1.el9 securityonion 346 k
procps-ng x86_64 3.3.17-14.0.1.el9 securityonion 382 k
python-unversioned-command noarch 3.9.18-3.el9 securityonion 8.3 k
python3 x86_64 3.9.18-3.el9 securityonion 30 k
python3-hawkey x86_64 0.69.0-8.0.1.el9 securityonion 110 k
python3-libdnf x86_64 0.69.0-8.0.1.el9 securityonion 806 k
python3-libs x86_64 3.9.18-3.el9 securityonion 9.6 M
python3-nftables x86_64 1:1.0.9-1.el9 securityonion 33 k
python3-perf x86_64 5.14.0-427.13.1.el9_4 securityonion 5.9 M
python3-pip-wheel noarch 21.2.3-7.el9_3.1 securityonion 1.1 M
python3-rpm x86_64 4.16.1.3-29.el9 securityonion 73 k
redhat-release x86_64 3:9.4-0.4.0.1.el9 securityonion 19 k
rpm x86_64 4.16.1.3-29.el9 securityonion 644 k
rpm-build-libs x86_64 4.16.1.3-29.el9 securityonion 88 k
rpm-libs x86_64 4.16.1.3-29.el9 securityonion 307 k
rpm-plugin-selinux x86_64 4.16.1.3-29.el9 securityonion 16 k
rpm-plugin-systemd-inhibit x86_64 4.16.1.3-29.el9 securityonion 16 k
rpm-sign-libs x86_64 4.16.1.3-29.el9 securityonion 20 k
rsyslog x86_64 8.2310.0-4.el9 securityonion 831 k
rsyslog-logrotate x86_64 8.2310.0-4.el9 securityonion 11 k
selinux-policy noarch 38.1.35-2.0.1.el9_4 securityonion 58 k
selinux-policy-targeted noarch 38.1.35-2.0.1.el9_4 securityonion 7.8 M
sqlite-libs x86_64 3.34.1-7.el9_3 securityonion 619 k
squashfs-tools x86_64 4.4-10.git1.el9 securityonion 175 k
sudo x86_64 1.9.5p2-10.el9_3 securityonion 1.1 M
systemd x86_64 252-32.0.2.el9_4 securityonion 4.6 M
systemd-libs x86_64 252-32.0.2.el9_4 securityonion 686 k
systemd-pam x86_64 252-32.0.2.el9_4 securityonion 282 k
systemd-rpm-macros noarch 252-32.0.2.el9_4 securityonion 75 k
systemd-udev x86_64 252-32.0.2.el9_4 securityonion 2.1 M
tcpdump x86_64 14:4.99.0-9.el9 securityonion 551 k
tzdata noarch 2024a-1.el9 securityonion 2.2 M
util-linux x86_64 2.37.4-18.0.1.el9 securityonion 2.5 M
util-linux-core x86_64 2.37.4-18.0.1.el9 securityonion 525 k
wireless-regdb noarch 2023.09.01-1.el9 securityonion 25 k
wpa_supplicant x86_64 1:2.10-5.el9 securityonion 1.6 M
xfsprogs x86_64 6.3.0-1.0.1.el9 securityonion 1.1 M
Installing group/module packages:
kernel-uek x86_64 5.15.0-205.149.5.1.el9uek securityonion 1.8 M
kernel-uek-core x86_64 5.15.0-205.149.5.1.el9uek securityonion 49 M
kernel-uek-modules x86_64 5.15.0-205.149.5.1.el9uek securityonion 51 M
Installing dependencies:
freetype x86_64 2.10.4-9.el9 securityonion 393 k
fuse3 x86_64 3.10.2-8.el9 securityonion 62 k
fuse3-libs x86_64 3.10.2-8.el9 securityonion 91 k
graphite2 x86_64 1.3.14-9.el9 securityonion 100 k
grub2-tools-efi x86_64 1:2.06-77.0.1.el9 securityonion 547 k
grub2-tools-extra x86_64 1:2.06-77.0.1.el9 securityonion 858 k
harfbuzz x86_64 2.7.4-10.el9 securityonion 632 k
kernel-core x86_64 5.14.0-427.13.1.el9_4 securityonion 21 M
kernel-modules x86_64 5.14.0-427.13.1.el9_4 securityonion 40 M
kernel-modules-core x86_64 5.14.0-427.13.1.el9_4 securityonion 35 M
libpng x86_64 2:1.6.37-12.el9 securityonion 117 k
openssl-fips-provider x86_64 3.0.7-2.0.1.el9 securityonion 574 k

Transaction Summary

Install 16 Packages
Upgrade 187 Packages

Total download size: 805 M
Downloading Packages:
(1/203): fuse3-libs-3.10.2-8.el9.x86_64.rpm 267 kB/s | 91 kB 00:00
(2/203): fuse3-3.10.2-8.el9.x86_64.rpm 182 kB/s | 62 kB 00:00
(3/203): freetype-2.10.4-9.el9.x86_64.rpm 918 kB/s | 393 kB 00:00
(4/203): graphite2-1.3.14-9.el9.x86_64.rpm 995 kB/s | 100 kB 00:00
(5/203): grub2-tools-efi-2.06-77.0.1.el9.x86_64 3.6 MB/s | 547 kB 00:00
(6/203): grub2-tools-extra-2.06-77.0.1.el9.x86_ 3.5 MB/s | 858 kB 00:00
(7/203): harfbuzz-2.7.4-10.el9.x86_64.rpm 1.7 MB/s | 632 kB 00:00
[MIRROR] kernel-modules-5.14.0-427.13.1.el9_4.x86_64.rpm: Status code: 404 for https://10.57.89.244/repo/kernel-modules-5.14.0-427.13.1.el9_4.x86_64.rpm (IP: 10.57.89.244)
[MIRROR] kernel-modules-5.14.0-427.13.1.el9_4.x86_64.rpm: Status code: 404 for https://10.57.89.244/repo/kernel-modules-5.14.0-427.13.1.el9_4.x86_64.rpm (IP: 10.57.89.244)
[MIRROR] kernel-modules-5.14.0-427.13.1.el9_4.x86_64.rpm: Status code: 404 for https://10.57.89.244/repo/kernel-modules-5.14.0-427.13.1.el9_4.x86_64.rpm (IP: 10.57.89.244)
[MIRROR] kernel-modules-5.14.0-427.13.1.el9_4.x86_64.rpm: Status code: 404 for https://10.57.89.244/repo/kernel-modules-5.14.0-427.13.1.el9_4.x86_64.rpm (IP: 10.57.89.244)
[FAILED] kernel-modules-5.14.0-427.13.1.el9_4.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
kernel-modules-5.14.0-427.13.1.el9_4.x86_64: Cannot download, all mirrors were already tried without success
2024-05-07T04:55:20Z | INFO | Removing repo files added by oracle-repos package update
2024-05-07T04:55:20Z | INFO | Executing command: rm -f /etc/yum.repos.d/oracle-linux-ol9.repo
2024-05-07T04:55:20Z | INFO | Executing command: rm -f /etc/yum.repos.d/uek-ol9.repo
2024-05-07T04:55:20Z | INFO | Executing command: rm -f /etc/yum.repos.d/virt-ol9.repo
2024-05-07T04:55:20Z | INFO | Installing Salt
2024-05-07T04:55:20Z | INFO | Executing command: dnf -y install salt-3006.5 salt-minion-3006.5
Security Onion Repo 14 kB/s | 3.0 kB 00:00
Package salt-3006.5-0.x86_64 is already installed.
Package salt-minion-3006.5-0.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
2024-05-07T04:55:21Z | INFO | Executing command: mkdir -p /etc/salt/minion.d
2024-05-07T04:55:21Z | INFO | Executing command: salt-pip install docker --no-index --only-binary=:all: --find-links files/salt_module_deps/docker/
Looking in links: files/salt_module_deps/docker/
Requirement already satisfied: docker in /opt/saltstack/salt/extras-3.10 (5.0.2)
Requirement already satisfied: websocket-client>=0.32.0 in /opt/saltstack/salt/extras-3.10 (from docker) (1.5.1)
Requirement already satisfied: requests!=2.18.0,>=2.14.2 in /opt/saltstack/salt/lib/python3.10/site-packages (from docker) (2.31.0)
Requirement already satisfied: certifi>=2017.4.17 in /opt/saltstack/salt/lib/python3.10/site-packages (from requests!=2.18.0,>=2.14.2->docker) (2023.7.22)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/saltstack/salt/lib/python3.10/site-packages (from requests!=2.18.0,>=2.14.2->docker) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in /opt/saltstack/salt/lib/python3.10/site-packages (from requests!=2.18.0,>=2.14.2->docker) (3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/saltstack/salt/lib/python3.10/site-packages (from requests!=2.18.0,>=2.14.2->docker) (1.26.18)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
2024-05-07T04:55:22Z | INFO | Executing command: salt-pip install pymysql --no-index --only-binary=:all: --find-links files/salt_module_deps/pymysql/
Looking in links: files/salt_module_deps/pymysql/
Requirement already satisfied: pymysql in /opt/saltstack/salt/extras-3.10 (1.0.3)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
2024-05-07T04:55:23Z | INFO | Configuring minion type as sensor
2024-05-07T04:55:23Z | INFO | Running: salt-call state.apply salt.mine_functions --local --file-root=../salt/ -l info pillar='{host: {mainint: eno1}}'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO ] Syncing grains for environment 'base'
[INFO ] Loading cache from salt://_grains, for base
[INFO ] Caching directory '_grains/' for environment 'base'
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', ** done ** 'salt/mine_functions.sls'
[INFO ] Running state [/etc/salt/minion.d/mine_functions.conf] at time 04:55:24.757508
[INFO ] Executing state file.managed for [/etc/salt/minion.d/mine_functions.conf]
[INFO ] Fetching file from saltenv 'base', ** done ** 'salt/etc/minion.d/mine_functions.conf.jinja'
[INFO ] File changed:
New file
[INFO ] Completed state [/etc/salt/minion.d/mine_functions.conf] at time 04:55:24.776043 (duration_in_ms=18.535)
local:

      ID: mine_functions
Function: file.managed
    Name: /etc/salt/minion.d/mine_functions.conf
  Result: True
 Comment: File /etc/salt/minion.d/mine_functions.conf updated
 Started: 04:55:24.757508
Duration: 18.535 ms
 Changes:
          ----------
          diff:
              New file
          mode:
              0644

Summary for local

Succeeded: 1 (changed=1)
Failed: 0

Total states run: 1
Total run time: 18.535 ms
2024-05-07T04:55:25Z | INFO | Not an appliance
2024-05-07T04:55:25Z | INFO | Executing command: salt-call state.apply setup.highstate_cron --local --file-root=../salt/
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO ] Syncing grains for environment 'base'
[INFO ] Loading cache from salt://_grains, for base
[INFO ] Caching directory '_grains/' for environment 'base'
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', ** done ** 'setup/highstate_cron.sls'
[INFO ] Running state [PATH=$PATH:/usr/sbin salt-call state.highstate] at time 04:55:26.643001
[INFO ] Executing state cron.present for [PATH=$PATH:/usr/sbin salt-call state.highstate]
[INFO ] Executing command 'crontab' in directory '/root'
[INFO ] Executing command 'crontab' in directory '/root'
[INFO ] {'root': 'PATH=$PATH:/usr/sbin salt-call state.highstate'}
[INFO ] Completed state [PATH=$PATH:/usr/sbin salt-call state.highstate] at time 04:55:26.657905 (duration_in_ms=14.904)
local:

      ID: post_setup_cron
Function: cron.present
    Name: PATH=$PATH:/usr/sbin salt-call state.highstate
  Result: True
 Comment: Cron PATH=$PATH:/usr/sbin salt-call state.highstate added to root's crontab
 Started: 04:55:26.643001
Duration: 14.904 ms
 Changes:
          ----------
          root:
              PATH=$PATH:/usr/sbin salt-call state.highstate

Summary for local

Succeeded: 1 (changed=1)
Failed: 0

Total states run: 1
Total run time: 14.904 ms
2024-05-07T04:55:26Z | INFO | Verifying setup
WARNING: Errors detected during setup.
--------- ERRORS ---------
[FAILED] kernel-modules-5.14.0-427.13.1.el9_4.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success

[root@aaoc-csoids-p01 ~]#'

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[reyesj2]

When you run dnf search --showduplicates kernel-modules what do you get from both the manager and the minion sensor?

[rosswakelin]

This is from the manager:
dnf search --showduplicates kernel-modules Last metadata expiration check: 0:04:24 ago on Fri 10 May 2024 01:42:20 AM UTC. ======================================== Name Exactly Matched: kernel-modules ======================================== kernel-modules-5.14.0-362.24.1.0.1.el9_3.x86_64 : kernel modules to match the core kernel kernel-modules-5.14.0-427.13.1.el9_4.x86_64 : kernel modules to match the core kernel kernel-modules-5.14.0-427.13.1.el9_4.x86_64 : kernel modules to match the core kernel ============================================ Name Matched: kernel-modules ============================================ kernel-modules-core-5.14.0-362.24.1.0.1.el9_3.x86_64 : Core kernel modules to match the core kernel kernel-modules-core-5.14.0-427.13.1.el9_4.x86_64 : Core kernel modules to match the core kernel kernel-modules-core-5.14.0-427.13.1.el9_4.x86_64 : Core kernel modules to match the core kernel kernel-modules-extra-5.14.0-362.18.0.2.el9_3.x86_64 : Extra kernel modules to match the kernel
This is from the sensor:
dnf search --showduplicates kernel-modules Last metadata expiration check: 2:57:54 ago on Thu 09 May 2024 10:47:42 PM UTC. ================================== Name Exactly Matched: kernel-modules ================================== kernel-modules-5.14.0-362.13.1.el9_3.x86_64 : kernel modules to match the core kernel kernel-modules-5.14.0-427.13.1.el9_4.x86_64 : kernel modules to match the core kernel ====================================== Name Matched: kernel-modules ====================================== kernel-modules-core-5.14.0-362.13.1.el9_3.x86_64 : Core kernel modules to match the core kernel kernel-modules-core-5.14.0-427.13.1.el9_4.x86_64 : Core kernel modules to match the core kernel kernel-modules-extra-5.14.0-362.18.0.2.el9_3.x86_64 : Extra kernel modules to match the kernel

[rosswakelin]

I know its bad form to answer your own question, but here goes.
We had this issue when the grid was at 2.4.60, and the "failing" setup was from a 2.4.40 iso install.
Yesterday we upgraded the grid to 2.4.70, and just for giggles I tried the setup again and it ran through and worked first time. Yah.
I can only presume that the combination of 2.4.60 and 2.4.40 was not happy, and 2.4.70 fixed it.