Elastic Status Flapping red / yellow / green

[craigsmooth]

Version

2.4.120

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

airgap

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

128 GB

Storage for /

5 TB

Storage for /nsm

10 TB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

My elasticsearch status / health seems to constantly be flapping between red, yellow, and green. Grid also coincides with "FAULT","PENDING","OK". Does anyone have any idea what might be causing the flapping of the cluster health? I have been periodically running _so-elasticsearch-query cluster/health?pretty to validate. Thanks for any tips.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[dougburks]

I have been periodically running _so-elasticsearch-query cluster/health?pretty to validate.

What exactly is this command telling you?

Do you have one or more search nodes? Are you having any network issues between the manager and the search nodes?

Have you checked the Elasticsearch log for additional clues?
https://docs.securityonion.net/en/2.4/elasticsearch.html#diagnostic-logging

[dougburks]

We don't have enough information to determine the cause. If it happens again, please provide the information requested above and we'll go from there.

[craigsmooth]

I am still seeing the same behavior with our cluster flapping between red, yellow, and green. I have included an excerpt of the elasticsearch log. Any ideas what could be the root issue here? Thank you.
securityonion_2025_03_11.log

[dougburks]

Looking at the log, I would focus on this:

reason: followers check retry count exceeded

So I'll go back to the question I asked previously:
Are you having any network issues between the manager and the search nodes?

Also see:
https://www.elastic.co/guide/en/elasticsearch/reference/8.17/troubleshooting-unstable-cluster.html#troubleshooting-unstable-cluster-follower-check

[craigsmooth]

Logs were certainly helpful. :) We were able to track it down to the cisco switch where the mgmt interfaces for the so nodes terminate. Switch was configured with storm-control which was rate limiting the traffic between the nodes causing network issues. Removing that seems to have resolved the issue. Thanks @dougburks