SO 2.3.2 - Distributed. Sensors not appearing

[szym3n]

Hi All,

Is there a good way to troubleshoot why sensors are not appearing in the GUI SO Interface?
It seems like there is data being sent over based on the networking traffic, but when I go into the GUI, I don't see the sensors listed (Except for one).

When I run sudo salt-key -P to list out the salt keys, I see the other sensors listed and accepted in there.

Thanks,
Szymon

[jertel]

The first place to check is the sensor's /opt/so/logs/sensoroni/sensoroni.log file to see why it can't connect to the manager. When a sensor is not listed on the sensors page it means that the sensor will not process any PCAP jobs that an analyst requested in Hunt or Alerts. This is different from the sensor sending its log metadata into the logstash pipeline.

[szym3n]

You're right, it seems like 'so.status' is not available on that sensor node I sent the logs for.

cagalseids03_sensor:
'so.status' is not available.
ERROR: Minions returned with non-zero exit code

Not sure what happened though as it was installed the same way that the other sensor node (which is working well).

[szym3n]

The /var/log/messages log shows these errors on that sensor node:
Nov 10 15:12:34 CAGALSEIDS03 su: (to ******) ******on pts/0
Nov 10 15:22:31 CAGALSEIDS03 salt-minion: [ERROR ] Function needs_restarting.check in mine_functions not available
Nov 10 15:38:13 CAGALSEIDS03 su: (to root) ******on pts/0
Nov 10 16:01:01 CAGALSEIDS03 systemd: Created slice User Slice of root.
Nov 10 16:01:01 CAGALSEIDS03 systemd: Started Session 21 of user root.
Nov 10 16:01:01 CAGALSEIDS03 systemd: Removed slice User Slice of root.
Nov 10 16:02:55 CAGALSEIDS03 systemd-logind: New session 22 of user ******.
Nov 10 16:02:55 CAGALSEIDS03 systemd: Started Session 22 of user ******.

[jertel]

I'm not well versed in the salt world but perhaps @m0duspwnens can give his two cents. He's a busy guy so you might be better off re-installing that node and watching the setup results more closely, in case something unexpected occurred.

[szym3n]

I think I might as well do a reinstall.
Odd that the salt-key was added to the master node and accepted.

[garthmahns]

I have the same issue, it says the sensor is trying to connect to my manger on its private IP address, which wont be possible since it is outside of our network and I am not using a VPN. I have tried changing the sensoroni config file IP address to the public IP, but upon restart it was overwritten back to the private IP

[szym3n]

This is what appears to be happening as I try to install again.
It's odd because I deleted the key out of the salt-key for this server prior to re-installing the node from the ISO.

Any idea what else I could have missed when cleaning up on the management?

[szym3n]

I tried doing the following:
Used kibana's dev tools to run:
PUT _cluster/settings
{
"persistent": {
"search": {
"remote": {
"CAGALSEIDS03_sensor": {
"skip_unavailable": null,
"seeds":null
}
}
}
}
}

PUT _cluster/settings
{
"persistent": {
"search": {
"remote": {
"cagalseids03_sensor": {
"seeds": null}}}}}

I also removed the old sensor's key from the master prior to this.

Same issue seems to happen regardless.

The SO Install/setup appears to be hanging on this screen for a bit of time, but when I run 'salt-key -P' as this installation is running, I DO NOT SEE any new 'unaccepted keys' in the list. The unaccepted key only shows up after the initial installation finishes and says that 'Install had a problem':

[szym3n]

Is anyone able to help out with this one?
Not sure what else to troubleshoot in this case.

[szym3n]

How do you go about clearing some of the old entries from the salt pillar?
When I run sudo salt-run cache.pillar
I have some old sensors that didn't get set up properly in there.
How do I clear that data?

[meredib]

What do you get when you run this command?
salt-key --list all

You may be able to simply delete the key from the list
salt-key -d <sensor_name>