Elasicsearch Faults on New Install

[Rennilon]

Version

2.4.40

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

135GB

Storage for /

135G

Storage for /nsm

8TB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

I have a new install that I then had to do a lot of work on to get RAID setup for nsm. Now I am getting faults due to elastic search on the manager and search nodes. Manger shows Elasticsearch Status: Fault and I see shard failure logs. Search node says the elasticsearch and logstash containers are missing. Process status shows fault and Elasticsearch shows as unknown.

On the search node. I see the following in the docker logs:
ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/securityonion.log

I'm a novice to this and trying to figure out how to troubleshoot the issue. Any help appreciated.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[Rennilon]

I tried restarting the logstash and elasticsearch containers. Both exit with code 1. In the logstash container logs I see the following:
Your settings are invalid. Reason: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.
Using bundled JDK: /usr/share/logstash/jdk

[Rennilon]

I fixed part of the issue. When I moved /nsm on the search node, I don't think I preserved file permissions. I modified /nsm/elasticsearch and /nsm/logstash and then the containers started and stay started. Search node looks green in the soc Grid screen now.

Manager still shows elasticsearch fault though so I wonder there are still permission issues I need to fix.

[Rennilon]

Clicking alert, dashboard, or hunt in the SOC gives me the following style error.

The search query encountered a failure within the Elasticsearch cluster. Check SOC logs for details.
See the Help section of the Security Onion documentation for additional troubleshooting guidance.
The Hunt interface may contain additional log messages to further diagnose the issue.

I see these in the sensaroni log:

{"fields":{"host":"sosearch01","processStatus":{"soman01":"red"}},"level":"warn","timestamp":"2024-02-06T21:54:25.198816743Z","message":"Host not found in process status metrics"}

{"fields":{"reason":"N/A","reasonType":"no_shard_available_action_exception"},"level":"warn","timestamp":"2024-02-06T21:32:39.741908409Z","message":"Shard failure"}

{"fields":{"error":"ERROR_QUERY_FAILED_ELASTICSEARCH", ... ,"level":"warn","timestamp":"2024-02-06T21:32:39.74206283Z","message":"Request did not complete successfully"}

[Rennilon]

I ended up rebuilding the servers. this time made sure to use cp -a when moving the /nsm files around. Things are all green now.