2.3.60 airgap salt-master could not be reached

[darth-drew]

2.3.60 airgap salt-master could not be reached

I have three separate distributed airgap installs with true clustering enabled. Each install has failed. For context, /root/soup.log indicates (shortly after copying Security Onion code from /tmp/agupdate/SecurityOnion to /opt/so/saltstack/default), that salt-master could not be reached. I am also unable to join any sensors to these managers, which sit at "accepting salt-key on the manager" before eventually failing.

All other upgrade attempts thus far have gone without issue. Is it possible that the airgap upgrade process for 2.3.60 (hotfix or initial) is broken? Should I revert back or is it possible someone smarter than myself can help me get this fixed?

[darth-drew]

I have just reproduced this error again, but from 2.3.50 to 2.3.61. Besides being airgapped and having true clustering enabled, these are essentially default, out of the box installs (from ISO) with no other configurations. There is plenty of hardware available. Any support would be greatly appreciated. At this point I am unable to demo the elastic authentication greatness.

[TOoSmOotH]

Is the salt master service running? Can you highstate on the manager?

[darth-drew]

Thanks for reaching out. so-salt-start is reporting the service has been up and running since the upgrade. I ran so-salt-stop and so-salt-start just in case but the sensor installs are still failing, even after what looks like a successful manager highstate.

[TOoSmOotH]

So the highstate on the manager is successful? @m0duspwnens Do you think this could be stuck in local only mode?

[m0duspwnens]

From the manager, can you check the status of the salt-master service with sudo systemctl status salt-master? If that is running, can you highstate the manager with sudo salt-call state.highstate -linfo?

For the sensors, are those new installs or are they preexisting installs that are now unable to communicate with the manager?

[darth-drew]

salt-master status is up for six days after I restarted last time. Highstate ran and indicated no issues (683 Succeded, 23 changed).. WRT the sensors, I am primarily doing new installs but have also tried Security-Onion/so-setup iso.. The issue is (at least the first time) the sensor cannot communicate with the salt master. I have had mixed success with re running setup, but I don't recall exactly what the errors were.

[darth-drew]

Correction, re-running setup seems to consistently give the same error, but occasionally it will still manage to download the correct containers and populate in grid/grafana. However, they also show as "fault" in grid even though so-status indicates all green for extended periods.

[wildlyunder]

Hi,

Did anyone find a fix for this?