Distributed Deployment - Salt Minion Errors

[dcarlin]

Version

2.4.150

Installation Method

Network installation on Red Hat derivative like Oracle, Rocky, Alma, etc. (unsupported)

Description

configuration

Installation Type

Distributed

Location

cloud

Hardware Specs

Meets minimum requirements

CPU

4

RAM

16

Storage for /

500

Storage for /nsm

500

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Good morning everyone. I'm working on a distributed deployment in AWS GovCloud deployed on RHEL 9.6 due to the unavailability of the Security Onion AMI in sub organization accounts. The issue I'm running into is that the sensor nodes install/configure/run flawlessly for ~ 1 day then all experience connectivity issues with reporting back to the managersearch instance.

When I re-run the command "sudo salt-call state.highstate" I receive the following error, indicating that the system cannot connect to the salt-master instance. Re installing security onion on the sensor nodes fixes this for another day. I'm at a loss for how to proceed and any guidance is appreciated.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

What is your Salt version for the master and minion? sudo salt-call --versions

[dcarlin]

Thank you for taking the time to respond.
Master: Salt: 3006.9
Minions: Salt: 3006.12

[m0duspwnens]

We had Salt 3006.12 in the repo to prepare for an upgrade, but we have decided to not upgrade due to a bug in Salt. As of today, that package has been removed from the repo. Even though 3006.12 was in the repo, it shouldn't have been installed. Were there any modifications that were made for the install that may have led to this? Can you give some detail on the install process?

In order to fix, all the minions will need to have 3006.9 installed. This include the manager since it runs both salt-master and salt-minion. According to your output, the salt-master is 3006.9, but the salt-minion service on the manager may be 3006.12. If Oracle, you should be able to use dnf to install salt-minion 3006.9.

[dcarlin]

That makes sense, I'll start by downgrading the minions. When I originally installed security onion from the github repo on top of RHEL 8 it didn't install, or at least successfully install, salt-minion on the system so I had to do a yum install salt-minion. This is likely the cause of the issue. I didn't think a minor mismatch of versions would cause an issue but I'm less shocked than I probably should be. I'll downgrade the versions and monitor.

[dcarlin]

Downgrading and modifying the security onion update window for minions has allowed the distribution to run successfully since the initial change. Closing this as resolved. Thank you.