[SOLVED] Cannot restart SecurityOnion services after blackout-reboot due to "Data failed to compile" error

[rgsax]

Version

2.4.110

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

40

RAM

37GB

Storage for /

65GB

Storage for /nsm

125GB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hi,
I'm having some issues after a reboot of the Security Onion virtual machines (manager + search + sensor + idh) due to a blackout.

Trying to restart the services the following error arises:

This is part of the content of /opt/so/log/salt/master (I couldn't copy/paste the content, I'm sorry):

I tried several commands, like "soup", "so-checkin" and "salt-call state.highstate -l info" but I always receved the same output.

Do you have any idea about how to fix this problem? All of the services are down and I don't know how to resolve.

Thanks in advance.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[zfcarsonb]

I am not a Security Onion maintainer, so take this with a grain of salt.

Based on the information you provided; it looks like your pillars or salt files may be corrupted. To the best of my knowledge, you basically have three options:

1. Perform a fresh install of Security Onion,
2. Attempt to recover your grid using a backup (see below)
3. (Expert) Locate and correct the corrupted pillar or salt files.

Attempt to restore from a backup

This is a last-resort option to try and recover your grid without re-installing.

Some warnings before proceeding:

• You may wish to wait for additional responses to your post, in case others have better solutions.
• I have not tested this procedure! Try at your own risk.
• Try this against a test cluster first before attempting in production!
• This will permanently overwrite your current grid configuration with the backup.

1. Read https://docs.securityonion.net/en/2.4/backup.html before proceeding.
2. Check if you have backups in /nsm/backup. The correct files should look like this:

3. Choose a backup from a last known good state, and copy it to the root directory:

user@managernode ~$ sudo cp /nsm/backup/so-config-backup-yyyy_mm_dd.tar /

4. Change to the root directory:

user@managernode ~$ cd /

5. Unpack the backup over your root directory.

user@managernode /$ sudo tar xvf so-config-backup-yyyy_mm_dd.tar

6. Wait at least 15-30 minutes before doing anything else for the restored salt state to apply.

All together:

user@managernode ~$ sudo cp /nsm/backup/so-config-backup-yyyy_mm_dd.tar /
user@managernode ~$ cd /
user@managernode /$ sudo tar xvf so-config-backup-yyyy_mm_dd.tar

[rgsax]

Thanks @zfcarsonb for your suggestion. Unfortunately that method didn't work for me.

I managed to resolve my issue in a different way.
Trying to restore the SecurityOnion backups didn't make the errors disappear.
After a long research, I found that the problem could be casued by some kind of problems in the salt-master cache.

The probem was solved after a cache clean of the salt master and a so-checkin.

Here are the step that I followed:

# systemctl stop salt-master
# rm -r /var/cache/salt/master
# systemctl start salt-master
# so-checkin

[zfcarsonb]

Glad to hear your grid is working again. Also, that's brilliant, I never would have thought to clear the salt cache to resolve this sort of issue. I may have to try that in the future if I run into similar problems.

[rgsax]

Just make sure to have time to spend for the activity. It took about 3 hours to set it up again. 😅