[2.3.0] Rendering exceptions during network installation of the sensor

[firion4ik]

Install source: Network installation.
CentOS 7
Sensor node
Does so-status show all services running? No

Hello!

I'm trying to add a new sensor to the existing 2.3.0 grid (was updated by soup).
The installation failed and I can see some rendering exceptions for a few components (see the extraction from the log file attached).
Would appreciate any help on this because I'm a bit stuck at the troubleshooting...

sensor_setup_errors.log

[firion4ik]

After I ran salt-call state.highstate I saw same "salt.exceptions.SaltReqTimeoutError: Message timed out" errors, but in the end there were just 5 failed components:

Summary for local
--------------
Succeeded: 171 (changed=14)
Failed:      5
--------------
Total states run:     176
Total run time:  1709.950 s

the so-status shows:

Checking Docker status

    Docker ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]   

Checking container statuses

    so-filebeat ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]   
    so-sensoroni ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]   
    so-steno ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]   
    so-strelka-coordinator -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]   
    so-strelka-filestream ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ ERROR ]   
    so-strelka-frontend -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-strelka-gatekeeper --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]   
    so-strelka-manager --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-suricata ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]   
    so-telegraf ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]   
    so-zeek ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]   

[TOoSmOotH]

Looks like the issue is with strelka. In /opt/so/saltstack/local/pillar/global.sls what does it have listed under strelka:

[firion4ik]

It has the following lines:

strelka:
  enabled: 1
  rules: 1

[firion4ik]

I tried to re-install the sensor (without re-installing the OS). Saw the same timeout errors and also
salt-minion[1894]: [WARNING ] Unable to send mine data to master.
in the journalctl -f output on the sensor during the re-install.
Same strelka components are showing ERROR in the so-status output after re-install.

Also, Grafana doesn't fetch statistics from this new sensor.

[firion4ik]

Hi again!
Would there be any suggestions on this?

[m0duspwnens]

@firion4ik
It seems as though there is some sort of connection issue from the sensor back to the master. Are there any network devices between he two that could be causing disruption?

Can you check iptables on the manager and confirm that the IP for the sensor is allowed to talk to ports 4505 and 4506? This rule should be created automatically during setup of the sensor, but I want to verify that it is there.

Do you have a search node? If so, does it have any issues when you run 'salt-call state.highstate'?

[firion4ik]

Hi @m0duspwnens
Yes, there is a firewall in between, but I can confirm that it doesn't block connections as I can successfully connect to 4505 and 4506 using nc.
On the manager node these ports are also opened

[root@securityonion-manager-node ~]# iptables -L -v -n | grep 450
 360K  152M ACCEPT     tcp  --  *      *       <sensor_ip>         0.0.0.0/0            tcp dpt:4506
 6243  407K ACCEPT     tcp  --  *      *       <sensor_ip>         0.0.0.0/0            tcp dpt:4505

I ran tcpdump during the setup and saw successful connections (requests and replies).
Also, during the setup I ran 'journalctl -f' command on the manager, it didn't show any packet drops by iptables.
It is weird because other components didn't have such an error. I ran the setup twice, and both times it was the same components which failed.

The 'salt-call state.highstate' output on the search node looks OK:

Summary for local
--------------
Succeeded: 183 (changed=6)
Failed:      0
--------------
Total states run:     183
Total run time:    69.908 s

[firion4ik]

Hi @m0duspwnens
Does the info above shed any light on the issue?

[firion4ik]

Hi!
Would there be any suggestions? Really looking forward to set it up and do performance testing before replacing the existing solution