SO-Elastalert Container Missing

[jaypfe]

Version

2.4.40

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

60

RAM

128

Storage for /

256

Storage for /nsm

256

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hello again! I am once again asking for you guys to descend some knowledge down upon me.

My SO-Elastalert container has gone missing. I have been fighting it for some time now, it happens almost after every reboot. But generally just asking it to restart, or .old-ing some of my elastalert rules fixes it. But this time, I have .old-ed all my rules. And have asked it to restart a great many times.

I am met with this response when running sudo so-elastalert-restart:
Server is not ready
Waiting for value 'green open' at 'https://localhost:9200/_cat /indices/.kibana*' (293/300)

It counts all the way up to 300 then:

---

      ID: so-elastalert
Function: docker_container.running
  Result: False
 Comment: One or more requisite failed: elastalert.enabled.wait_for_elastics                                                                                                                                                                                                                                                                                                                                                           earch
 Started: 15:12:39.281767
Duration: 0.003 ms
 Changes:

---

      ID: delete_so-elastalert_so-status.disabled
Function: file.uncomment
    Name: /opt/so/conf/so-status/so-status.conf
  Result: True
 Comment: Pattern already uncommented
 Started: 15:12:39.281862
Duration: 2.524 ms
 Changes:

Summary for local
Succeeded: 12 (changed=2)
Failed: 2

Running a high state provides just:
local:
Data failed to compile:
The function "state.highstate" is running as PID 2453618 and was started at 2024, Feb 09 15:13:07.834643 with jid 20240209151307834643

The elastalert.log in /opt/so/log is empty.
The kibana.log file only shows errors on some kibana rules I am running (the default ones from Elastic). These have been running for some months now without issue.

For my elastalert rules, I have most contained within the playbook folder. I have one contained outside of the playbook folder. Not sure if that matters for anything.

Really appreciate any kind of help or guidance on this issue!

Thanks again!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Check your Elasticsearch log - /opt/so/log/elasticsearch/ sounds like there is an issue connecting to Elasticsearch.

[cm-ops]

Still sounds like the issue is connecting to Elasticsearch. Is your cluster all the way up? so-elasticsearch-query _cluster/health?pretty

[jaypfe]

Yeah showing "Red" Status.

All other containers show to be up on all other nodes. I have 2 search nodes, the manager, a fleet node, a forward node, a receiver, with 1 IDH.

"cluster_name" : "securityonion",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 378,
"active_shards" : 378,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 80,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 82.53275109170306

I can still query elastic in the SOC and Kibana.

I am pretty lost to be honest with this one honestly ha.

[jaypfe]

I catted the securityonion.log file in /opt/so/log/elasticsearch and grepped for error and got about a million of theseL
Search rejected due to missing shards [[.ds-logs-endpoint.events.network-default-2024.02.13-000055][0]]. Consider using allow_partial_search_results setting to bypass this error.

I dont seem to be getting logs in anymore from endpoints, / not being able to index them. Have a shard warning in Kibana.

[samuel809]

check heap memory config

[jaypfe]

That did it, I threw a few more gigs at it. Thanks for that!

Confused though as it is only a problem now after 4 months of operation.

[jtchi717]

I seem to be having the same issue... how did you "check heap memory config"? or how did you resolve the issue?
Thanks.

[jaypfe]

I also did the workaround to make sure indices were not being stored on the man node, not sure if that helped as well.

If you just search for 'Heap' in the admin area, under Elasticsearch its the 'esheap' option.

I turned my rules back on, and its missing again today so pretty sure one of those are causing an issue as well.

[nebriv]

Hey @jaypfe, you mentioned this:

I also did the workaround to make sure indices were not being stored on the man node, not sure if that helped as well.

What work around is this?