Shards failing and now logstash and elasticsearch will not start

[sikoraj]

Version

2.4.60

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

Varies depending on the node; minimum of 4

RAM

Again varies depending on the node, minimum of 16 GB

Storage for /

Varies, minimum of 128 GB

Storage for /nsm

Varies heavily, between 500 GB and 70 TB

Network Traffic Collection

span port

Network Traffic Speeds

more than 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

We have been working on building out a new distributed 2.4 install. We have VMs for the manager node and two search nodes and we have physical servers for two forward sensor nodes and one physical search node box. We connected our first forward node to capture traffic from our datacenter to the rest of our campus and everything seemed to work fine for about a week.

Then we started to get faults on the manager node and the error "The search query encountered a failure within the Elasticsearch cluster. " when trying to look at alerts. Looking at the logs and in Kibana under Stack Management we found that we had shards failing and that some Zeek indices from the forward node were showing red under health. We tried deleting the indices that were having difficulty, and that fixed the issue temporarily. Shortly after the indices would show red and the shards would fail again.

Now after rebooting the entire system the forward node is now also showing a fault and that elasticsearch and logstash are missing. Trying to restart elasticsearch and logstash keeps failing.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[reyesj2]

From the manager can you give us the output from something like
sudo salt -C 'G@role:so-searchnode or G@role:so-sensor' state.highstate -linfo

[sikoraj]

Thanks!
I have attached the results to this post
results.txt

[reyesj2]

On sostorep1 can you run the command
test -d /nsm && echo "correct" | echo "incorrect" /nsm should be a directory and it looks like currently it might exist as a file ? If the output says "incorrect" try removing the file

On sostorev1 it looks like it might be having problems communicating with the manager check that it still has access to the manager. You can try running nc -zv <managerip> 4505 & nc -zv <managerip> 4506 Here is a list of ports that all nodes should back to the manager https://docs.securityonion.net/en/2.4/firewall.html#node-communication

[sikoraj]

Thank you for the help!

Running this revealed that the file system for nsm on our one search node was corrupted.

We worked with one of our sys admins and we were able to run xfs_repair several times with different options in order to repair it. Once repaired and rebooted it came back up and has now been working.