Elasticsearch crashing after a few hour

[JhonShell]

Version

2.4.80

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

512

Storage for /

100

Storage for /nsm

1tb

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hi,
I am currently having issue with security onion the elasticsearch service is clashing after a few hour , I suspentest that is due all the index open
log

[root@search01 elasticsearch]# docker logs so-elasticsearch
Importing PKCS12 keypair into Java keystore
Importing keystore /usr/share/elasticsearch/config/elasticsearch.p12 to /usr/share/elasticsearch/config/sokeys...
Entry for alias 1 successfully imported.
Import command completed: 1 entries successfully imported, 0 entries failed or cancelled
Sep 16, 2024 5:22:36 PM sun.util.locale.provider.LocaleProviderAdapter
WARNING: COMPAT locale provider will be removed in a future release
java.lang.OutOfMemoryError: Java heap space
Dumping heap to data/java_pid98.hprof ...
Heap dump file created [3293271948 bytes in 59.921 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[JhonShell]

After increasing the HEAP of the search node above the recommend (16 gb where 10gb to elasticsearch heap) , Also I decreased the sharps limit to 25.

[dougburks]

Installation Type
Distributed

Since you specified that this is a distributed installation, was Elasticsearch crashing on one of your search nodes?

RAM
512

After increasing the HEAP of the search node above the recommend (16 gb where 10gb to elasticsearch heap)

Are you saying your search node has 512GB RAM or 16GB RAM?

If your search node has 16GB RAM, please be advised that that is considered the bare minimum:
https://docs.securityonion.net/en/2.4/hardware.html#minimum-specs
https://docs.securityonion.net/en/2.4/hardware.html#search-node

Depending on how much data you are ingesting and searching, you may need more RAM. Or you may need to do more tuning for heap and shards:
https://docs.securityonion.net/en/2.4/elasticsearch.html#heap-size
https://docs.securityonion.net/en/2.4/elasticsearch.html#shards

[JhonShell]

I meant 16gb of ram on the search node.

[dougburks]

As I mentioned in my last reply, if your search node has 16GB RAM, please be advised that that is considered the bare minimum:
https://docs.securityonion.net/en/2.4/hardware.html#minimum-specs
https://docs.securityonion.net/en/2.4/hardware.html#search-node

Depending on how much data you are ingesting and searching, you may need more RAM. Or you may need to do more tuning for heap and shards:
https://docs.securityonion.net/en/2.4/elasticsearch.html#heap-size
https://docs.securityonion.net/en/2.4/elasticsearch.html#shards

[JhonShell]

now after adjusting the sharp and the heap , I have the issue:
[2024-09-25T23:55:42,250][INFO ][logstash.outputs.elasticsearch] Retrying failed action {:status=>429, :action=>["create", {:_id=>nil, :_index=>"logs-elastic_agent.endpoint_security-default", :routing=>nil}, {"agent"=>{"type"=>"endpoint", "version"=>"8.10.4", "id"=>"9b4bd1e4-5b20-45dd-ae18-ff63c5b544d0", "ephemeral_id"=>"51e3b6c8-5bb5-4694-a312-8f86b47a8b7b", "name"=>"hostname135"}, "process"=>{"pid"=>4120, "thread"=>{"id"=>6612}}, "message"=>"BulkQueueConsumer.cpp:197 Logstash connection is down", "metadata"=>{"input"=>{"beats"=>{"host"=>{"ip"=>"10.3.1.194"}}}, "input_id"=>"filestream-monitoring-agent", "raw_index"=>"logs-elastic_agent.endpoint_security-default", "type"=>"_doc", "stream_id"=>"filestream-monitoring-endpoint-default", "beat"=>"filebeat", "version"=>"8.10.4"}, "data_stream"=>{"type"=>"logs", "dataset"=>"elastic_agent.endpoint_security", "namespace"=>"default"}, "type"=>"redis-input", "tags"=>["elastic-agent", "input-receiver01", "beats_input_codec_plain_applied"], "@timestamp"=>2024-09-25T22:47:19.291Z, "@Version"=>"1", "ecs"=>{"version"=>"1.11.0"}, "input"=>{"type"=>"filestream"}, "elastic_agent"=>{"id"=>"9b4bd1e4-5b20-45dd-ae18-ff63c5b544d0", "snapshot"=>false, "version"=>"8.10.4"}, "event"=>{"dataset"=>"elastic_agent.endpoint_security"}, "log"=>{"origin"=>{"file"=>{"line"=>197, "name"=>"BulkQueueConsumer.cpp"}}, "file"=>{"vol"=>1726968908, "path"=>"C:\Program Files\Elastic\Endpoint\state\log\endpoint-000000.log", "idxlo"=>94921, "idxhi"=>786432}, "level"=>"notice", "source"=>"endpoint-default", "offset"=>15740800}, "host"=>{"hostname"=>"sdg-d-135", "ip"=>["fe80::4bc4:9806:7474:9048", "10.3.1.194"], "mac"=>["D0-8E-79-0C-AA-3E"], "id"=>"2ad5b31c-6178-4a1f-a141-849d5cd24b4a", "name"=>"sdg-d-135", "os"=>{"platform"=>"windows", "kernel"=>"10.0.22621.4169 (WinBuild.160101.0800)", "name"=>"Windows 11 Pro", "family"=>"windows", "type"=>"windows", "build"=>"22631.4169", "version"=>"10.0"}, "architecture"=>"x86_64"}, "component"=>{"type"=>"endpoint", "id"=>"endpoint-default", "binary"=>"endpoint-security", "dataset"=>"elastic_agent.endpoint_security"}}], :error=>{"type"=>"circuit_breaking_exception", "reason"=>"[parent] Data too large, data for [indices:data/write/bulk[s]] would be [5345885134/4.9gb], which is larger than the limit of [5259657216/4.8gb], real usage: [5345880736/4.9gb], new bytes reserved: [4398/4.2kb], usages [inflight_requests=5368/5.2kb, request=0/0b, fielddata=2842/2.7kb, eql_sequence=0/0b, model_inference=0/0b]", "bytes_wanted"=>5345885134, "bytes_limit"=>5259657216, "durability"=>"TRANSIENT"}}
[2024-09-25T23:55:42,250][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request {:count=>4}

the sharps number of sharps is set to 1 that mean that only one sharp would be create for an index and now I am having the error above.

[dougburks]

If you're not able to tune your heap and shards, then you may need more than the bare minimum of 16GB RAM.

[JhonShell]

I see that the issue was because all the traffic on queue due the downtime , I would like to know if there is a way to avoid it because now it seems to be ok, with 2 shards and Elasticsearch 9830m and Logstash 4096m , at the beginning it was thawing the exception however after 8 hour it get stable.