Distributed Deployment - Manager Fault Status

[NeoLife-iT]

Version

2.4.150

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

256

Storage for /

512

Storage for /nsm

1TB

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

I've been clearing the data and re-install + setup the cluster multiple time in the last 4 weeks - learned a lot from doing so.
After moving the Data Role off the Manager node - the grid show manager fault - reboot does not help -

How do i get rid of this error?

Thanks for all your help

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[Ximalas]

I have never changed the datarole for any node, but I do run this command on the manager after every reboot:

sudo so-elasticsearch-query _cluster/settings -d '{ "transient" : { "cluster.routing.allocation.exclude._ip" : "ip-address-of-manager" } }' -XPUT

By now I could probably use the persistent section:

sudo so-elasticsearch-query _cluster/settings -d '{ "persistent" : { "cluster.routing.allocation.exclude._ip" : "ip-address-of-manager" } }' -XPUT

[NeoLife-iT]

in had this Kilbana interface before reboot:

I ran this @ the console now:

sudo so-elasticsearch-query _cluster/settings -d '{ "persistent" : { "cluster.routing.allocation.exclude._ip" : "10.2.9.99" } }' -XPUT

Thanks for replying

still stuborn! error

[cm-ops]

Any clues in the /opt/so/log/elasticsearch/securityonion.log?

[NeoLife-iT]

Found the clue in the indice health check. now I need to remove them...i m very close to solved this problem..

This screen shot show my current ill cluster

[NeoLife-iT]

This is a sample good indice...newly setup cluster...show no red shards...i am setup for comparison and learning

[NeoLife-iT]

Delete the un-assigned index...It worked!!!!!!!!

[NeoLife-iT]

[NeoLife-iT]

Not sure why this error trigger on this search node...plenty of memory and cpu and Hardisk space

[NeoLife-iT]

i found 17 un-assign indexes that are hidden and can not be delete. How to fix this?

[NeoLife-iT]

Remove the bad node from the cluster and re-install and add it back. The cluster still fault because of the bad indices not sure how to remove them

[NeoLife-iT]

Clear all indices -- We're all good now -

[NeoLife-iT]

I now have detection error -

[Ximalas]

Try this, I'm writing from memory, so the details might be a bit fuzzy: Click on Rule Mismatch, try to determine the identifying rule number, go back to Detections, search for the offending rule, if said rule is disabled, enable it, wait a few seconds, disable it again, and run a differential update or full update on Suricata.

[NeoLife-iT]

When i click on the rule mismatch it took me here..show nothing

when i just got to dectection with 24 time frame it showed me bunch of mis-match...so where to find the number?

Thanks for reply

[NeoLife-iT]

-changed the date to 1 month i can see the mis-match rules - looking for info on how to remediate this issue...Tips? Look like i have 100 of these rules mis-match

[Ximalas]

Recently, I had issues with deactivated Suricata rules. I reactivated and then deactivated them, ran an update, and no more errors. I'm not sure if this applies in your case. If you haven't deactivated any rules, why not deactivate the lot of them, reactivate them in the next instance, and run an update?

[NeoLife-iT]

I'm keeping this distributed cluster for training and upgrading test run . I switched to Stand A Lone installation for 1 cluster - Same host and network segment so far no error and everything seem to be fine without the PCAP feature.

Thanks for all your help.