sensor and search node health in distributed deployment

[bretthanus]

I have a distributed deployment with a mgr node, 3 search nodes, and 5 forward nodes.
the managernode seems pretty solid on 2.3.70.
`Checking Docker status

Docker ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]

Checking container statuses

so-aptcacherng --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-cortex -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-dockerregistry ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-elastalert ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-elasticsearch ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-filebeat ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-fleet --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-grafana ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-idstools ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-influxdb ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-kibana -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-kratos -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-logstash ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-mysql --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-nginx --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-playbook ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-redis --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-sensoroni ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-soc ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-soctopus ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-telegraf ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-thehive ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-thehive-es ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-wazuh --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]

`

but my search nodes, one of the three seems to get hung/become unresponsive every 2-3 days. to resolve this i have to reset in vcenter. I'd like to review logs to figure out where my issues are, but i am unsure as to where i should look.

i also have one forward node of the five, that is in the second most active network seeing around 200-300 EPS, reported in Grid as Fault, and upon checking
`
Checking Docker status

Docker ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]

Checking container statuses

so-filebeat ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-nginx --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-sensoroni ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-steno --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-strelka-backend ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ PENDING ]
so-strelka-coordinator ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-strelka-filestream -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-strelka-frontend ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-strelka-gatekeeper -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-strelka-manager ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-suricata ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-telegraf ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-wazuh --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-zeek ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]

`
the strelka-backend remains in pending for a long while- I don't know how to get it started without a reboot of the box.

sorry to bother, TIA
brett

[dougburks]

I'd like to review logs to figure out where my issues are, but i am unsure as to where i should look.

Have you checked the documentation?
https://docs.securityonion.net/en/2.3/filebeat.html#diagnostic-logging
https://docs.securityonion.net/en/2.3/logstash.html#log
https://docs.securityonion.net/en/2.3/redis.html#diagnostic-logging
https://docs.securityonion.net/en/2.3/elasticsearch.html#diagnostic-logging