Upgrade from 2.4.50 to 2.4.100 failing (soup failed with error 23: Unhandled error)

[fa1sepr0phet]

Version

2.4.50

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Standalone

Location

airgap

Hardware Specs

Exceeds minimum requirements

CPU

6

RAM

128GB

Storage for /

293GB

Storage for /nsm

5.8TB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

When attempting to upgrade to 2.4.100, soup is failing with an unknown error. See below for soup.log output. once it fails, so-status shows so-dockerregistry is "missing" for about 10 minutes then returns to a "running" state. When I ran salt-cal state.highstate, it seemed to be successful but returned with two failures (could not scroll up to see where it errored out) then attempted to run it again,it returns with the error "The function "state.highstate" is running as PID 3729689 and was started at (today's date) with jid (long number).
Anyway, I have attempted the fixes:
#12078

#12635

from the SecurityOnion documentation and neither fix solved the issue. Looking for additional assistance with where I could start looking to troubleshoot the issue so I can get the upgrade going.

Last login: Mon Sep 16

Last metadata expiration check: 0:01:23 ago on Mon 16 Sep 2024 04:58:54 PM UTC.

Package salt-minion-3006.9-0.x86_64 is already installed.

Dependencies resolved.

Nothing to do.

Complete!

• INFO: Running install_oracle_linux_onedir_post()

• INFO: Running install_oracle_linux_check_services()

• INFO: Salt installed! (0)

Applying yum versionlock for Salt.

Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 04:58:54 PM UTC.

Adding versionlock on: salt-0:3006.9-0.*

Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 04:58:54 PM UTC.

Adding versionlock on: salt-minion-0:3006.9-0.*

Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 04:58:54 PM UTC.

Adding versionlock on: salt-master-0:3006.9-0.*

Checking if Salt was upgraded.

Salt upgrade success.

Checking to see if changes are needed.

Creating directory to store Suricata classification.config

Stopping Playbook services & cleaning up...

so-playbook

so-mysql

so-soctopus

Playbook Migration is complete....

sending incremental file list

sent 92 bytes received 12 bytes 208.00 bytes/sec

total size is 0 speedup is 0.00

IDStools configuration has been backed up.

sending incremental file list

sent 19 bytes received 12 bytes 62.00 bytes/sec

total size is 0 speedup is 0.00

Starting crond service at 17:00:27.181815

Successfully started crond.

Starting salt-master service at 17:00:27.225682

Successfully started salt-master.

Starting salt-minion service at 17:00:28.015802

Successfully started salt-minion.

Enabling highstate.

local:

----------

msg:

    Info: highstate state enabled.

res:

    True

Soup failed with error 23: Unhandled error

Unhandled error occured, please check /root/soup.log for details.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

What does the /root/soup.log show?

[fa1sepr0phet]

Please see the attached (sanitized) soup log.
soup.txt