Unhandled error when upgrading to 2.3.190

[kiaraho]

Ran soup on Standalone manager running 2.3.110 that was previously upgraded from 2.3.100 back in March. Job fails with an unhandled error. All services in the green, however when I log into SOC the node is still running 2.3.110. Ran salt-call state.highstate which completed successfully, though nothing changed.

The attached soup.log does not provide much from the line starting at ### Preparing soup at Sat Dec 24 00:51:49 UTC 2022 ###. Only error I found after this line was Minions returned with non-zero exit code and salt jid 20221224013212021349 which later completed.

The manager node is a VM so I reset it back to a snapshot I took before running soup. Same outcome on three separate tries. Any pointers would be helpful.

[kiaraho]

So after multiple failed upgrade attempts, I gave up and installed a fresh 2.3.190 instance. Luckily this deployment is being used for class instruction, so the data loss was inconsequential. Still, it would be good to know what went wrong.

[dougburks]

Looking at your soup.log, it shows this nears the end:

Stopping thehive...

So I suspect the issue had to do with upgrading a really old version that had thehive enabled. We haven't supported thehive since we released Security Onion 2.3.120 in April of last year:
https://blog.securityonion.net/2022/04/security-onion-23120-now-available.html

[kiaraho]

The original install on this instance was 2.3.100. The most recent upgrade was 2.3.110, after which the instance remained offline for several months. Is the upgrade from 2.3.110 to 2.3.190 not supported?

As stated earlier, this instance was used for educational purposes, so wiping it was not a big deal. However, we have more critical production instances that started off on earlier versions of 2.3.x, and have been upgraded more frequently. What upgrade paths are not supported?

[dougburks]

In theory, upgrading from 2.3.110 to 2.3.190 should work correctly. It's possible that there was some underlying issue with your thehive configuration that prevented it from working correctly.

In general, we recommend keeping up to date with releases as they fix bugs and resolve security issues.

[dougburks]

I happened to run into this issue today. I put in a fix for the upcoming 2.3.200 release so this shouldn't be an issue for anybody after that release.